Tests as gates

Where you are

Lesson 2's pipeline only printed messages. This lesson swaps them for real checks: a linter, unit tests, and integration tests that talk to a real Postgres database and a real Redis cache, the same mix CivicGrid's pipeline runs. You'll build it in your cicd-sandbox repository and run it in both GitHub Actions and Jenkins.

Jenkins needs one upgrade first: the ability to start containers of its own, which this lesson and the next both rely on. Set aside about 45 minutes.

A good gate

A check is only worth having as a gate if it's:

• Fast. If the checks take half an hour, people stop waiting for them, merge anyway, or batch changes up, which brings back Lesson 1's big scary merges. Minutes, not hours.
• Reliable. It passes when the code is fine and fails when it isn't, every time. A flaky test, one that fails now and then for no reason in the code, is the toast alarm: people learn to press re-run without reading the failure, and one day the failure is real.
• Meaningful. It actually checks something that matters, and it actually runs. A test that never runs, or runs but checks nothing, is the dead battery.

Keep those three in mind. Every part of this lesson is about one of them.

Three kinds of check

Most pipelines run three kinds of check, from quickest to slowest:

• Linting reads the code without running it, looking for style problems and common mistakes, like an import that's never used or a variable that's never defined. It takes seconds and catches a surprising number of real bugs. CivicGrid uses Ruff, a fast Python linter.
• Unit tests run small pieces of code on their own, like one function, with nothing else involved. They're fast and pinpoint exactly what broke.
• Integration tests check that pieces work together, including real services like a database. They're slower and need more setup, but they catch the mistakes unit tests can't, like a SQL query that's valid Python but wrong SQL.

A healthy pipeline has lots of the first two and enough of the third to trust the parts that touch the outside world. CivicGrid's integration tests run against a real Postgres, because a fake one would happily accept queries the real one rejects.

stage('Quality Gate') {
  steps {
    timeout(time: 10, unit: 'MINUTES') {
      script {
        def qualityGate = waitForQualityGate()
        if (qualityGate.status != 'OK') {
          error("Quality Gate failed: ${qualityGate.status}")
        }
      }
    }
  }
}

The app you'll test

Your sandbox gets a tiny app that counts page visits: it keeps a running count in Redis and records each visit in Postgres. It's small, but it has everything a real pipeline needs: code to lint, logic to unit test, and two services to integration test against.

On your laptop, create the app and its tests in your sandbox. Every command in this lesson runs in that folder unless it says otherwise, so start there. If you run these in another repository by mistake, they'll add files to it. First the app:

cd ~/Apps/cicd-sandbox
mkdir -p app tests
touch app/__init__.py
cat > app/visits.py <<'EOF'
def clean_name(name):
    """Trim spaces and lowercase a page name, so 'Home ' and 'home' count as one page."""
    cleaned = name.strip().lower()
    if not cleaned:
        raise ValueError("page name can't be empty")
    return cleaned


def record_visit(db, cache, name):
    """Count a visit in Redis and record it in Postgres. Returns the new count."""
    page = clean_name(name)
    count = cache.incr(f"visits:{page}")
    with db.cursor() as cur:
        cur.execute("INSERT INTO visits (page) VALUES (%s)", (page,))
    db.commit()
    return count
EOF

Then the tests. The first two are unit tests: they only need Python. The third is an integration test: it needs a real database and cache, and it skips itself if it isn't told where they are, the same pattern CivicGrid's tests use:

cat > tests/test_visits.py <<'EOF'
import os

import psycopg2
import pytest
import redis

from app.visits import clean_name, record_visit

DB_URL = os.environ.get("TEST_DATABASE_URL")
REDIS_URL = os.environ.get("REDIS_TEST_URL")


def test_clean_name_trims_and_lowercases():
    assert clean_name("  Home ") == "home"


def test_clean_name_rejects_empty():
    with pytest.raises(ValueError):
        clean_name("   ")


@pytest.mark.skipif(not (DB_URL and REDIS_URL), reason="TEST_DATABASE_URL and REDIS_TEST_URL not set")
def test_record_visit_counts_and_stores():
    db = psycopg2.connect(DB_URL)
    cache = redis.Redis.from_url(REDIS_URL)
    with db.cursor() as cur:
        cur.execute("DROP TABLE IF EXISTS visits")
        cur.execute("CREATE TABLE visits (id serial PRIMARY KEY, page text NOT NULL)")
    db.commit()
    cache.delete("visits:home")

    assert record_visit(db, cache, "Home") == 1
    assert record_visit(db, cache, "home ") == 2

    with db.cursor() as cur:
        cur.execute("SELECT count(*) FROM visits WHERE page = 'home'")
        assert cur.fetchone()[0] == 2
    db.close()
EOF

And two small settings files. pytest.ini tells pytest to look for the app folder from the top of the repository, and requirements-dev.txt lists what the tests need, pinned to exact versions so every run installs the same thing:

cat > pytest.ini <<'EOF'
[pytest]
pythonpath = .
testpaths = tests
EOF
cat > requirements-dev.txt <<'EOF'
pytest==9.0.3
psycopg2-binary==2.9.13
redis==8.1.0
EOF

Notice the integration test starts by dropping and recreating its table, and clearing its Redis key. That's what makes it reliable: it never depends on what an earlier run left behind. It's also why it must only ever point at a throwaway database. CivicGrid's integration tests refuse to run against anything but localhost for the same reason: they drop the whole schema.

In GitHub Actions: service containers

GitHub can start extra containers alongside a job, called service containers, and throw them away when the job ends. That's exactly what an integration test wants: a brand new Postgres and Redis for every run, with nothing left over from the last one.

Replace Lesson 2's workflow with a real one. This is CivicGrid's pipeline in miniature:

git rm -q .github/workflows/first.yml
mkdir -p .github/workflows
cat > .github/workflows/ci.yml <<'EOF'
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  contents: read

jobs:
  lint:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: '3.13'
      - run: pip install ruff
      - run: ruff check .

  test:
    runs-on: ubuntu-24.04
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: visits_test
        ports:
          - 5432:5432
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10
      redis:
        image: redis:7.2.7-alpine
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10
    env:
      TEST_DATABASE_URL: postgresql://postgres:postgres@localhost:5432/visits_test
      REDIS_TEST_URL: redis://localhost:6379/0
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: '3.13'
      - run: pip install -r requirements-dev.txt
      - run: pytest -q -rs

  build:
    needs: [lint, test]
    runs-on: ubuntu-24.04
    steps:
      - run: echo "Would build an image for run ${{ github.run_number }}"
EOF

The mkdir -p line is there because git rm removed the only file in .github/workflows, and Git deletes folders that end up empty, so the folder has to be made again before writing into it. I found that out the hard way.

The new parts, all from CivicGrid's real ci.yml:

• services: starts postgres:17 and redis:7.2.7-alpine next to the test job. env: under postgres sets the database's password and name, the way you'd set them with docker run -e.
• ports: publishes each service on the runner, so the tests reach them at localhost:5432 and localhost:6379.
• options: passes Docker health checks. GitHub waits until pg_isready and redis-cli ping say yes before starting the job's steps. Without this, the tests can start before Postgres is ready to accept connections and fail for no reason in the code: a flaky test you built yourself.
• The job's env: tells the tests where the services are, which is what stops the integration test skipping itself.
• pytest -q -rs runs the tests quietly, and -rs adds a short report of any skipped tests and why. You'll see why that matters at the end.
• permissions: contents: read keeps the job's token read-only. Lesson 2's log showed the token's permissions; this writes them down instead of relying on the repository's default. Lesson 5 explains it.

The password postgres is fine here, because the database only exists for a few minutes, on a machine nobody else can reach, with nothing in it. It's not a real secret. Lesson 5 covers the ones that are.

In Jenkins: Docker agents and sidecars

Jenkins has no built-in service containers. Instead, you give Jenkins access to Docker, and the Jenkinsfile starts the containers itself. Two ideas do all the work:

• A Docker agent runs a stage inside a container from an image you choose, like python:3.13-slim. Jenkins mounts the workspace into it, so the stage sees the code. That's how you get Python without installing it on the agent.
• A sidecar is a container that runs alongside the stage for as long as it needs, like a database for the tests. The Docker Pipeline plugin's withRun starts one, hands you its ID, and removes it afterwards, which is Jenkins' version of a service container.

Here's the Jenkinsfile. Save it at the top of your sandbox repository:

cat > Jenkinsfile <<'EOF'
pipeline {
  agent any
  options {
    timeout(time: 15, unit: 'MINUTES')
  }
  environment {
    PIP_DISABLE_PIP_VERSION_CHECK = '1'
    PIP_NO_CACHE_DIR = '1'
  }
  stages {
    stage('Checks') {
      parallel {
        stage('Lint') {
          agent {
            docker {
              image 'python:3.13-slim'
              reuseNode true
            }
          }
          steps {
            sh '''
              python -m venv .venv-lint
              . .venv-lint/bin/activate
              pip install --quiet ruff
              ruff check .
            '''
          }
        }
        stage('Tests') {
          steps {
            script {
              docker.image('postgres:17').withRun('-e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=visits_test') { pg ->
                docker.image('redis:7.2.7-alpine').withRun() { rd ->
                  docker.image('postgres:17').inside("--link ${pg.id}:db") {
                    sh 'until pg_isready -h db -U postgres; do sleep 1; done'
                  }
                  docker.image('python:3.13-slim').inside("--link ${pg.id}:db --link ${rd.id}:cache") {
                    sh '''
                      python -m venv .venv-test
                      . .venv-test/bin/activate
                      pip install --quiet -r requirements-dev.txt
                      export TEST_DATABASE_URL=postgresql://postgres:postgres@db:5432/visits_test
                      export REDIS_TEST_URL=redis://cache:6379/0
                      pytest -q -rs
                    '''
                  }
                }
              }
            }
          }
        }
      }
    }
    stage('Build') {
      steps {
        echo "Would build an image for build ${env.BUILD_NUMBER}"
      }
    }
  }
}
EOF

Reading it:

• agent { docker { image 'python:3.13-slim'; reuseNode true } } runs the Lint stage in a Python container. reuseNode true keeps it on the same agent and workspace as the rest of the pipeline, so it sees the code Jenkins checked out.
• withRun starts Postgres and Redis as sidecars. The { pg -> ... } block is the time they're alive; when it ends, Jenkins stops and removes them, even if a test failed.
• inside("--link ${pg.id}:db") runs a container that can reach the Postgres sidecar by the name db. The first one uses Postgres' own pg_isready in a loop until the database answers: the same job as GitHub's health check, written by hand.
• The script { } block is needed because withRun and inside come from Jenkins' older scripted style. Declarative pipelines allow a little scripted code inside script, which is common in real Jenkinsfiles.
• timeout stops a stuck build after 15 minutes instead of letting it hang forever, a habit worth having in every pipeline.
• '''triple quotes''' let one sh step run several lines, so the virtual environment it activates stays active for the commands after it.

And one detail that only exists because of the van. Both stages make a Python virtual environment, a folder with its own installed packages, and they make it in the shared workspace, at the same time, because they run in parallel. If both used the folder name .venv, they'd install into the same folder at once and trip over each other. So they're .venv-lint and .venv-test. On GitHub, each job has its own machine, so the problem can't happen.

--link is an older Docker feature that newer setups replace with user-defined networks, but it's still what the Jenkins handbook's own sidecar examples use, and it keeps this file short.

Give Jenkins Docker

The Jenkins from Lesson 2 can't start containers: there's no Docker inside its container. The Jenkins handbook's answer is to run Docker in a second container next to it, called Docker in Docker (often written DinD), and point Jenkins at it. You'll rebuild your Jenkins this way, keeping the jenkins_home volume, so your admin user and first-pipeline job stay.

Why not just let Jenkins use Docker Desktop directly? You could, by sharing Docker Desktop's control socket with the container, and plenty of tutorials do. But anything that can talk to that socket can control every container on your laptop, and mount any of your files into one. A separate Docker for Jenkins keeps its builds away from everything else you run.

All of this is on your laptop. First, a network for the two containers to talk on, and the Docker container itself:

docker network create jenkins
docker run -d --name jenkins-docker --privileged --network jenkins --network-alias docker --env DOCKER_TLS_CERTDIR=/certs -v jenkins-docker-certs:/certs/client -v jenkins_home:/var/jenkins_home --restart=on-failure docker:dind

• docker:dind is the official Docker in Docker image: a whole Docker engine in a container.
• --privileged gives it the extra rights a Docker engine needs. That's a lot of power, which is exactly why it's a separate container just for Jenkins, not exposed to anything else.
• --network-alias docker lets Jenkins reach it by the name docker.
• DOCKER_TLS_CERTDIR=/certs makes it create certificates, so only clients with the right one, Jenkins, can control it. The client certificate goes in the jenkins-docker-certs volume.
• -v jenkins_home:/var/jenkins_home mounts Jenkins' own volume into it, at the same path. When Jenkins asks this Docker to mount a workspace like /var/jenkins_home/workspace/sandbox into a container, that path has to exist on this side too.
• No -p at all: nothing on your laptop, or your network, can reach this Docker engine. Only containers on the jenkins network can.

Next, a Jenkins image with the Docker command-line tool and the Docker Pipeline plugin added. Make a folder for it outside your repositories:

mkdir -p ~/Apps/jenkins-image
cd ~/Apps/jenkins-image
cat > Dockerfile <<'EOF'
FROM jenkins/jenkins:lts-jdk21
USER root
RUN apt-get update \
 && apt-get install -y ca-certificates curl \
 && install -m 0755 -d /etc/apt/keyrings \
 && curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc \
 && chmod a+r /etc/apt/keyrings/docker.asc \
 && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" > /etc/apt/sources.list.d/docker.list \
 && apt-get update \
 && apt-get install -y docker-ce-cli \
 && rm -rf /var/lib/apt/lists/*
USER jenkins
RUN jenkins-plugin-cli --plugins docker-workflow
EOF
docker build -t jenkins-with-docker .

The RUN line adds Docker's own package source and installs only the command-line tool, docker-ce-cli, not a Docker engine. jenkins-plugin-cli installs the Docker Pipeline plugin, whose ID is docker-workflow, which is what gives Jenkinsfiles agent { docker ... }, withRun and inside. It switches back to the jenkins user at the end, so Jenkins doesn't run as root.

Now swap the old Jenkins container for the new one. Removing the container doesn't touch the jenkins_home volume, which is where everything you set up lives:

docker rm -f jenkins
docker run -d --name jenkins --network jenkins -p 127.0.0.1:8080:8080 -v jenkins_home:/var/jenkins_home -v jenkins-docker-certs:/certs/client:ro --env DOCKER_HOST=tcp://docker:2376 --env DOCKER_CERT_PATH=/certs/client --env DOCKER_TLS_VERIFY=1 --restart=on-failure jenkins-with-docker

The three DOCKER_ settings point Jenkins' Docker tool at the jenkins-docker container, over an encrypted connection, using the certificate from the shared volume, mounted read-only (:ro). Check that it worked:

docker exec jenkins docker version

Mine, trimmed to the parts that matter:

Client: Docker Engine - Community
 Version:           29.9.2
 API version:       1.56
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.9.2
  API version:      1.56 (minimum version 1.40)
  OS/Arch:          linux/arm64

Two sections, two programs. Client is the docker command you installed in the Jenkins image. Server is the engine in jenkins-docker, answering over the network. If Jenkins couldn't reach it, you'd get the Client section followed by an error instead of a Server section, which makes this a quick check whenever builds can't start containers. linux/arm64 is because my laptop has an Apple processor, and Docker runs Linux containers in a small virtual machine on it; on most other computers it would say linux/amd64. Lesson 4 comes back to why that difference matters for images.

A good place to stop. Jenkins now has everything this lesson and the next need, and it keeps it all in its volumes. If you're taking a break, docker stop jenkins jenkins-docker pauses both, and docker start jenkins-docker jenkins brings them back later, Docker engine first.

Run it in both tools

GitHub first. On your laptop, in ~/Apps/cicd-sandbox:

git add -A
git status --short
git commit -m "Visit counter with lint, unit and integration tests"
git push

git status --short should list the app, the tests, the two settings files, the Jenkinsfile and the new workflow, and show first.yml as deleted. Give it a minute, then:

gh run list --limit 1
gh run view "$(gh run list --limit 1 --json databaseId --jq '.[0].databaseId')" --log | grep -E -i "passed|skipped|failed|healthy"

The second line is long because of one detail: when gh run view's output goes into another command, like grep here, it can't ask which run you mean, so it needs the run's ID. The part in $( ) asks gh run list for the newest run's ID and fills it in. grep then keeps only the lines about the services' health and the test results.

Mine:

lint Run ruff check . 2026-10-25T13:50:19.0820194Z All checks passed!
test Initialize containers 2026-10-25T13:50:33.6834022Z healthy
test Initialize containers 2026-10-25T13:50:33.6852576Z postgres service is healthy.
test Initialize containers 2026-10-25T13:50:33.6987266Z healthy
test Initialize containers 2026-10-25T13:50:33.7002369Z redis service is healthy.
test Run pytest -q -rs 2026-10-25T13:50:41.0890879Z 3 passed in 0.54s

Ruff found nothing. In the test job, GitHub's Initialize containers step waited for both health checks before any of the job's steps ran, and about eight seconds later, after checking out the code and installing the requirements, pytest ran. 3 passed is the line to look for: all three tests, including the integration test, which means it found the database and the cache and didn't skip. The whole run took 43 seconds.

Now Jenkins. Jenkins needs to read the Jenkinsfile from your repository, and your sandbox is private, so Jenkins would need a GitHub credential to clone it. Credentials are Lesson 5. Until then, the simplest way is to make the sandbox public: it holds nothing but this lesson's practice files.

gh repo edit --visibility public --accept-visibility-change-consequences

Then in Jenkins, at http://localhost:8080:

1. New Item, name it sandbox, choose Pipeline, OK.
2. Under Pipeline, set Definition to Pipeline script from SCM, and SCM to Git.
3. Repository URL: https://github.com/Kwamib/cicd-sandbox.git, or your own sandbox's address.
4. Branch Specifier: */main. Script Path: Jenkinsfile, which is the default.
5. Save, then Build Now.

The first build is slow: the Docker engine inside jenkins-docker starts empty, so it downloads python:3.13-slim, postgres:17 and redis:7.2.7-alpine before anything runs. Later builds reuse them.

My first build's log was over 300 lines, most of it image downloads. These are the lines worth finding in yours, in the order they appeared:

Obtained Jenkinsfile from git https://github.com/Kwamib/cicd-sandbox.git
No credentials specified
Checking out Revision 2be56acd83d606a990aad4a007a495f1afeb0cbb (refs/remotes/origin/main)
+ docker inspect -f . python:3.13-slim

error: no such object: python:3.13-slim
+ docker run -d -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=visits_test postgres:17
Unable to find image 'postgres:17' locally
+ docker pull python:3.13-slim
Jenkins seems to be running inside container b42c788d200dff01c906c75758594a2c1c0922eaa37e2aac9f0754749bdcb9cc
but /var/jenkins_home/workspace/sandbox could not be found among []
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/sandbox -v /var/jenkins_home/workspace/sandbox:/var/jenkins_home/workspace/sandbox:rw,z -v /var/jenkins_home/workspace/sandbox@tmp:/var/jenkins_home/workspace/sandbox@tmp:rw,z -e ******** -e ******** python:3.13-slim cat
+ ruff check .
All checks passed!
+ docker run -d redis:7.2.7-alpine
+ pg_isready -h db -U postgres
db:5432 - accepting connections
+ pytest -q -rs
...                                                                      [100%]
3 passed in 0.11s
+ docker stop f436eb7cfa88efded709e2943a403aac918e08cf58afcd70c06c8049e2f7a1d8
+ docker rm -f --volumes f436eb7cfa88efded709e2943a403aac918e08cf58afcd70c06c8049e2f7a1d8
+ docker stop 7c6a34640a4a8009db856e7f67d5d6df4f7a50cecab28ec64b0ece674a955602
+ docker rm -f --volumes 7c6a34640a4a8009db856e7f67d5d6df4f7a50cecab28ec64b0ece674a955602
Would build an image for build 1
Finished: SUCCESS

From the top:

• Obtained Jenkinsfile from git and No credentials specified: Jenkins read the pipeline from your repository, which it can do without a credential because the repository is public. The Declarative: Checkout SCM stage then cloned the exact commit you pushed.
• error: no such object: python:3.13-slim looks alarming, but it's Jenkins checking whether the image is already there before pulling it. On the first build it never is.
• Jenkins seems to be running inside container ... could not be found among [] is Jenkins noticing it's in a container and looking for a way to share its folders with the new one. It doesn't find one, because the Docker engine is a different container, so it falls back to mounting the workspace by its path with -v. That only works because jenkins-docker has the jenkins_home volume at the same path, which is why the setup mounted it.
• docker run ... -u 1000:1000 ... python:3.13-slim cat is the Docker agent. -u 1000:1000 runs it as the same user as Jenkins, not as root; -v mounts the workspace; and cat is a trick that keeps the container waiting while Jenkins runs the stage's commands inside it. The -e ******** values are environment settings, which Jenkins hides in the log in case any of them are secret.
• db:5432 - accepting connections is the wait loop succeeding on its first try, and 3 passed is all three tests, against the sidecars.
• The docker stop and docker rm -f --volumes pairs are withRun removing Redis, then Postgres, once their block ended. Nothing is left behind for the next build.

Your log will also have about twenty lines of shell detail after each . .venv-…/bin/activate. That's the + tracing from Lesson 2 showing every line of the activation script; it's harmless.

Now the Jenkinsfile lives in your repository, next to the code it checks. Change one, and the history shows both: that's pipeline as code.

Open and close the gate

Lesson 1 said the build only starts if the checks pass. That's true because of one line, needs: [lint, test], and it's surprisingly easy to undo by accident. Fail a check, then try the three ways a build job is commonly wired:

if: always() deserves its own warning. It's genuinely useful: a step that uploads test reports, posts a message or cleans up should run whether the tests passed or not. Jenkins puts that kind of step in post { always { ... } }. The mistake is putting it on the build or deploy job, usually while trying to fix something else, like "the build job keeps getting skipped". The skip was the gate working.

Break it on purpose

Time to watch real checks fail. Each of these is a one-line change, and each fails a different kind of check.

A lint failure. Add an import that's never used to the top of app/visits.py:

sed -i '' '1i\
import os
' app/visits.py
head -3 app/visits.py
git commit -am "Add an unused import"
git push

Run Build Now in Jenkins too. On GitHub, gh run view with --log-failed prints only the steps that failed. Here's the Ruff part of mine:

gh run view "$(gh run list --limit 1 --json databaseId --jq '.[0].databaseId')" --log-failed

I001 [*] Import block is un-sorted or un-formatted
 --> app/visits.py:1:1
  |
1 | import os
  | ^^^^^^^^^
2 | def clean_name(name):
  |
help: Organize imports

F401 [*] `os` imported but unused
 --> app/visits.py:1:8
  |
1 | import os
  |        ^^
2 | def clean_name(name):
  |
help: Remove unused import: `os`

Found 2 errors.
[*] 2 fixable with `--fix`.
##[error]Process completed with exit code 1.

I expected one error and got two. F401 is the one I planted: os is imported and never used. I001 is a bonus: Python style puts two blank lines between the imports and the first function, and my sed line put none. Each error names the rule, the file and the line, points at the exact spot with ^^^, and suggests a fix. [*] 2 fixable with --fix means ruff check --fix would correct both for you, which is handy on your own computer. In the pipeline, the job should only report: a check that quietly rewrites code isn't a gate any more.

Why two? The workflow runs pip install ruff with no version, so it gets the newest Ruff, and newer versions check more rules by default. Mine checked import formatting; an older one might not have. A tool whose version you don't control can change what your gate checks without anyone changing your code. Lesson 6 pins it, and CivicGrid's own ci.yml has the same unpinned line.

Jenkins doesn't know about the push, so press Build Now yourself; Lesson 8 sets up the automatic part. Its Lint stage found the same two errors, because it also installs the newest Ruff. The end of mine:

Found 2 errors.
[*] 2 fixable with `--fix`.
Failed in branch Lint
+ pytest -q -rs
...                                                                      [100%]
3 passed in 0.12s
Stage "Build" skipped due to earlier failure(s)
ERROR: script returned exit code 1
Finished: FAILURE

Notice the Tests branch kept going after Lint failed and passed all three tests, because the two run side by side and nothing told Tests to stop. The Build stage, which waits for both, was skipped. If you'd rather stop everything at the first failure, to save time on a long pipeline, Jenkins has failFast true for parallel blocks, and GitHub cancels a job matrix's other jobs by default. Either way, the gate is the same: nothing gets built.

Put it back before the next one:

sed -i '' '1d' app/visits.py
git commit -am "Remove the unused import"

A logic failure. Make clean_name forget to lowercase:

sed -i '' 's/cleaned = name.strip().lower()/cleaned = name.strip()/' app/visits.py
git commit -am "Stop lowercasing page names"
git push

Lint has nothing to complain about, because the code is perfectly valid Python. The unit test catches it: clean_name(" Home ") now returns "Home". So does the integration test, because Home and home are now counted as two pages.

Here's GitHub's, from --log-failed, with the timestamps taken off and the middle of the second test cut short:

F.F                                                                      [100%]
=================================== FAILURES ===================================
_____________________ test_clean_name_trims_and_lowercases _____________________

    def test_clean_name_trims_and_lowercases():
>       assert clean_name("  Home ") == "home"
E       AssertionError: assert 'Home' == 'home'
E         
E         - home
E         ? ^
E         + Home
E         ? ^

tests/test_visits.py:14: AssertionError
_____________________ test_record_visit_counts_and_stores ______________________
        assert record_visit(db, cache, "Home") == 1
>       assert record_visit(db, cache, "home ") == 2
E       AssertionError: assert 1 == 2

tests/test_visits.py:33: AssertionError
2 failed, 1 passed in 0.30s

How to read it:

• F.F is one character per test, in order: failed, passed, failed. The one that passed is test_clean_name_rejects_empty, because the bug doesn't touch empty names.
• The > line is the exact line that failed, and the E lines say why. pytest shows both sides: it got 'Home' and expected 'home', and the ? and ^ marks point at the one character that differs.
• The integration test failed in a more telling way. The first visit to Home returned 1, as expected. The second visit, to home , also returned 1, not 2, because without lowercasing, Home and home became two different pages with a count each. The unit test says the function is wrong; the integration test says what that would do to real data.
• tests/test_visits.py:14 gives the file and line number, so you can go straight there.

One more thing in the full log: the test job's settings showed TEST_DATABASE_URL: ***localhost:5432/visits_test, with the username and password replaced by ***. GitHub's runner hides passwords it recognises inside addresses like this one. That's helpful, but it's a safety net, not a plan: it can't recognise every secret, which is why Lesson 5 is about keeping them out of logs in the first place.

Put it back:

sed -i '' 's/cleaned = name.strip()/cleaned = name.strip().lower()/' app/visits.py
git commit -am "Lowercase page names again"
git push

The green pipeline that tested nothing. This is the one to remember. Delete the env: block from the test job in ci.yml, the two lines that tell the tests where Postgres and Redis are, and push:

sed -i '' '/^    env:$/,/^      REDIS_TEST_URL/d' .github/workflows/ci.yml
grep -n "TEST_DATABASE_URL" .github/workflows/ci.yml
git commit -am "Lose the test database settings"
git push

grep should print nothing: the settings are gone. The pipeline goes green. Every job passes. But look at the test job's output:

gh run view "$(gh run list --limit 1 --json databaseId --jq '.[0].databaseId')" --log | grep -E -i "passed|skipped"

test Run pytest -q -rs 2026-10-25T14:08:24.1359709Z SKIPPED [1] tests/test_visits.py:22: TEST_DATABASE_URL and REDIS_TEST_URL not set
test Run pytest -q -rs 2026-10-25T14:08:24.1360519Z 2 passed, 1 skipped in 0.37s
lint Run ruff check . 2026-10-25T14:08:01.3781799Z All checks passed!

The integration test skipped itself, the database and cache sat there unused, and the run still passed. Nothing in the summary turns red. That's the dead smoke alarm: a gate that looks switched on and isn't. The -rs report is the only clue, which is why it's in the command.

Put it back with git revert --no-edit HEAD and git push, which undoes the last commit by making a new one.

Spot the problem

The Monday morning test. A test checks that a report says "due this week". It passes all week, then fails every Monday, and passes again if someone re-runs it in the afternoon. Nobody's changed the code. What's going on? The test depends on the real date and time, so its result depends on when it runs, not on the code. That's flaky, and it trains people to press re-run. The fix is to give the code a fixed date in the test instead of asking the clock.

Two builds, one database. A Jenkins team keeps one long-running Postgres server for all their test builds, to save start-up time. Tests pass on their own, but when two builds run at the same time, they sometimes fail with duplicate rows and missing tables. Why? Both builds share the same database, so one build's tests drop and recreate tables while the other's are using them. Throwaway sidecars, one set per build, would keep them apart. That's what withRun gives you, and what GitHub's service containers give every job automatically.

The test that can't fail. A test calls record_visit and then assert True. It's been green for months. Is it a good test? It's barely a test at all. It checks that the function doesn't crash, and nothing about what it does. If record_visit returned the wrong count or wrote nothing to the database, this would still pass. A useful test asserts the result, like == 2, and that the side effect happened, like the row count.

Where you are now

Your sandbox now has a real pipeline in both tools: Ruff for linting, unit tests for the logic, and integration tests against a fresh Postgres and Redis every run. GitHub gave each test job its own databases with services: and health checks; Jenkins started them as sidecars with withRun, waited with pg_isready, and ran everything in Docker containers on its own Docker engine. You've seen needs: hold the gate and if: always() open it, and you've seen the quietest failure of all: a skipped test that left the pipeline green. Lesson 4 replaces the last echo: building a real container image, tagging it, and pushing it to a registry, in both tools.

Check yourself

Key terms

• Gate: a check that must pass before the next stage runs.
• Flaky test: a test that sometimes fails for reasons that have nothing to do with the code.
• Linting: checking code for problems without running it.
• Unit test: a test of one small piece of code on its own.
• Integration test: a test that checks pieces working together, often with real services like a database.
• Service container: a container GitHub runs alongside a job, like a database, and removes afterwards.
• Health check: a command Docker runs to see whether a container is ready, like pg_isready.
• Docker agent: a Jenkins stage that runs inside a container from a chosen image.
• Sidecar: a container that runs alongside a build step for as long as it's needed.
• Docker in Docker (DinD): a Docker engine running inside a container.
• Skipped test: a test that didn't run, often because something it needs is missing.

Go deeper

• GitHub's guide to creating PostgreSQL service containers (opens in a new tab), which covers running a job directly on the runner, as here, or inside a container of its own.
• The Jenkins handbook's Using Docker with Pipeline (opens in a new tab), including its sidecar examples.
• Ruff's list of rules (opens in a new tab), to see what F401 and the other default checks look for.