CodeBuild test reports: run tests, view results (JUnit)

The builds in previous articles "succeeded", but that success only says the commands ran without returning an error — it does not say the code is correct. That is the difference between "the build runs" and "the software works". The "CI" in CI/CD includes running tests every time code is integrated. This article has CodeBuild run real tests and emit a test report — a report that collects test results so you can see at a glance how many tests passed, how many failed, and where. This article also closes Part II.

Goal

Have CodeBuild run a test suite and collect the results into a viewable report, understand what a report group is, and know how to make a failing test block the build.

Add tests to the app

The demo app is a static page, but there is still plenty worth checking: the file exists, the content is right, there is a version marker. Write a test suite with pytest, in tests/test_app.py:

import os, re

def test_index_exists():
    assert os.path.isfile("index.html"), "index.html must exist"

def test_index_has_heading():
    html = open("index.html", encoding="utf-8").read()
    assert "Hello from the awscicd demo app" in html

def test_version_marker():
    html = open("index.html", encoding="utf-8").read()
    assert re.search(r"v\d+", html), "expected a version marker like v2"

Declare tests and report in the buildspec

Two changes in buildspec.yml: run the tests in the build phase (emitting a JUnit XML file), and add a reports block telling CodeBuild to collect that file into a report:

phases:
  install:
    commands:
      - pip install -q pytest
  build:
    commands:
      - echo "Running tests"
      - pytest tests/ --junitxml=report.xml -v

reports:
  demo-tests:
    files:
      - "report.xml"
    file-format: JUNITXML

pytest --junitxml=report.xml runs the tests and writes results to report.xml in JUnit format — the standard format that nearly every test framework (JUnit, pytest, Jest, Go test...) can emit. The reports block with file-format: JUNITXML tells CodeBuild to read that file and build it into a report. CodeBuild also understands other formats (NUNITXML, TESTNGXML, CUCUMBERJSON, and code coverage).

Grant the role report permissions

Building a report needs more permissions for the service role — create and update report groups, write each test case:

{
  "Effect": "Allow",
  "Action": ["codebuild:CreateReportGroup","codebuild:CreateReport",
             "codebuild:UpdateReport","codebuild:BatchPutTestCases","codebuild:BatchPutCodeCoverages"],
  "Resource": "arn:aws:codebuild:*:*:report-group/awscicd-demo-build-*"
}

Without these permissions, the build still runs but the report-creation part fails. The resource is scoped exactly to the project's report group (<project>-*).

Run and view the results

After pushing and running the build, each build is associated with a report. Get the report ARN from the build, then view the summary:

$ RARN=$(aws codebuild batch-get-builds --ids "$BID" --query 'builds[0].reportArns[0]' --output text)
$ aws codebuild batch-get-reports --report-arns "$RARN" \
    --query 'reports[0].[status,testSummary.total,testSummary.statusCounts]'
[
    "SUCCEEDED",
    3,
    { "ERROR": 0, "FAILED": 0, "SKIPPED": 0, "SUCCEEDED": 3, "UNKNOWN": 0 }
]

Three tests, all three passed. View each case:

$ aws codebuild describe-test-cases --report-arn "$RARN" \
    --query 'testCases[].[name,status,durationInNanoSeconds]' --output text
test_index_exists       SUCCEEDED  0
test_version_marker     SUCCEEDED  1000000
test_index_has_heading  SUCCEEDED  1000000

This is the value of a report: instead of wading through thousands of log lines to find which test failed, you see straight away a list of tests with their status and per-test duration. When a test fails, it shows FAILED with the name, leading you straight to what needs fixing.

   buildspec: pytest --junitxml=report.xml
        │
        ▼  CodeBuild reads report.xml (JUnit)
   ┌──────────── Report group: awscicd-demo-build-demo-tests ───────────┐
   │  report (this build): 3 total, 3 SUCCEEDED                          │
   │   ├─ test_index_exists        SUCCEEDED                             │
   │   ├─ test_index_has_heading   SUCCEEDED                             │
   │   └─ test_version_marker      SUCCEEDED                             │
   │  ... each build adds a report → pass/fail trend over time          │
   └─────────────────────────────────────────────────────────────────────┘

Report group: tracking over time

Note the report group name: awscicd-demo-build-demo-tests (the project name plus the report name you declared). A report group collects every report of the same kind across builds, so you can see trends — how the pass/fail ratio changes over time, which tests fail often. Each build creates a new report in that group.

Make a failing test block the build

An easy mistake with a big impact. If you write the test command as pytest ... || true, the build will always "succeed" even when tests fail — the report still records the failures, but the build is green, and broken code moves on to deploy. That defeats the purpose of CI. The right way is to let the test command return its natural error code: pytest tests/ --junitxml=report.xml (no || true). When a test fails, pytest returns a non-zero code, the build phase fails, the build fails, and the pipeline (Part V) stops right there — broken code cannot slip through to production. The report lets you see where it failed; the non-zero exit code blocks it from moving on.

🧹 Cleanup

The report group is created automatically; delete it when cleaning up the series:

$ aws codebuild delete-report-group \
    --arn arn:aws:codebuild:ap-southeast-1:111122223333:report-group/awscicd-demo-build-demo-tests \
    --delete-reports

Reports and report groups incur no meaningful storage cost, so keeping them across articles is fine.

Wrap-up

A test report turns "the build ran without error" into "knowing exactly which tests passed/failed". Have CodeBuild run tests emitting JUnit XML (pytest --junitxml), then declare a reports block with file-format: JUNITXML; grant the role permission to create report groups and write test cases. Results are viewable via batch-get-reports (total/pass/fail) and describe-test-cases (each case), collected in a report group that tracks trends. Important: do not swallow test errors with || true — let a failing test fail the build so CI actually blocks broken code.

Part II closes here: we have built, passed secrets safely, and run tests with reports. Part III is a short but useful supporting service — CodeArtifact, an internal package repository — so builds pull and publish private libraries instead of always going out to the Internet.

CodeBuild Test Reports: A Build Must Not Only Run, It Must Be Correct