Pipelines that still work next month

Where you are

Your pipelines work. Both tools lint, test against real databases, build an image and push it with properly kept secrets. This lesson adds nothing new to what they do. It's about what happens to a working pipeline over the next few months, as tools release new versions, people push in a hurry and the disk slowly fills.

Each section starts with something that breaks, then the setting that stops it. None of them is a new tool, just a few lines in files you already have.

A car that passed its test

Green that means nothing: the skip trap

What breaks. In Lesson 3, deleting two lines from ci.yml made the integration test skip itself, and the pipeline stayed green with 2 passed, 1 skipped. Nobody would notice. CivicGrid has 13 test files built the same way, so the same tidy-up would silently stop testing its database code.

The fix is a conftest.py, a file pytest loads by itself before any test runs. It stops the run with an error when it's in CI and the settings the integration tests need are missing. GitHub sets CI=true on every runner; your laptop doesn't, so the tests still skip politely there, which is what you want. On your laptop, in ~/Apps/cicd-sandbox:

cd ~/Apps/cicd-sandbox
cat > tests/conftest.py <<'EOF'
import os

import pytest


def pytest_configure(config):
    if os.environ.get("CI") and not (os.environ.get("TEST_DATABASE_URL") and os.environ.get("REDIS_TEST_URL")):
        raise pytest.UsageError(
            "Running in CI without TEST_DATABASE_URL and REDIS_TEST_URL: "
            "the integration tests would skip, and the run would pass without testing them."
        )
EOF
git add tests/conftest.py
git commit -m "Fail in CI when the integration tests would skip"

Now repeat Lesson 3's mistake on purpose, and push both commits together:

sed -i '' '/^    env:$/,/^      REDIS_TEST_URL/d' .github/workflows/ci.yml
grep -n "TEST_DATABASE_URL" .github/workflows/ci.yml
git commit -am "Lose the test database settings, again"
git push
sleep 10
gh run watch "$(gh run list --limit 1 --json databaseId --jq '.[0].databaseId')" --exit-status
gh run view "$(gh run list --limit 1 --json databaseId --jq '.[0].databaseId')" --log-failed | grep -E "ERROR|Error|exit code"

This time the run goes red at the test job, and build never starts. Mine, with the timestamps taken off:

test Run pytest -q -rs ERROR: Running in CI without TEST_DATABASE_URL and REDIS_TEST_URL: the integration tests would skip, and the run would pass without testing them.
test Run pytest -q -rs ##[error]Process completed with exit code 4.

The message says exactly what's wrong, in words the next person will understand. Exit code 4 is pytest's code for "I was run wrongly", which keeps it apart from a test that failed (code 1). Put the settings back:

git revert --no-edit HEAD
git push

The next run is back to normal, with the check still in place for next time:

test Run pytest -q -rs 3 passed in 1.19s

Pushes that pile up: concurrency

What breaks. You push a fix, spot a typo, and push again, then once more. GitHub starts three full runs, side by side. Two of them are testing code that's already out of date, and you pay runner time for all three. On a deploy pipeline it's worse: two runs can deploy at once and finish in the wrong order, leaving the older code live.

A concurrency group says "only one run of this at a time". Add one to the top of ci.yml, plus a ten-minute limit on every job, which the next section explains:

python3 - <<'EOF'
from pathlib import Path
p = Path(".github/workflows/ci.yml")
s = p.read_text()
top = "permissions:\n  contents: read\n\njobs:\n"
assert s.count(top) == 1 and s.count("    runs-on: ubuntu-24.04\n") == 3
s = s.replace(top, "permissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: ${{ github.event_name == 'pull_request' }}\n\njobs:\n")
s = s.replace("    runs-on: ubuntu-24.04\n", "    runs-on: ubuntu-24.04\n    timeout-minutes: 10\n")
p.write_text(s)
EOF
git diff

The new block:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

• group names the queue. github.workflow and github.ref make it "this workflow, on this branch", so each pull request and main get a queue of their own.
• One running, one waiting. Inside a group, only one run goes at a time and only one more can wait. A newer push replaces whatever was waiting.
• cancel-in-progress decides whether a newer push also stops the run that's already going. Here it's only on pull requests, where an old run is pure waste. On main, the running build might be halfway through pushing an image, and cutting that off is worse than a short wait.

Try the three settings for yourself. Each run takes 55 seconds, like the sandbox's:

Then see it for real. Commit the change and push two empty commits straight after it:

git commit -am "Replace superseded runs and cap each job at 10 minutes"
git push
git commit --allow-empty -m "Concurrency test 1"
git push
git commit --allow-empty -m "Concurrency test 2"
git push
sleep 10
gh run watch "$(gh run list --limit 1 --json databaseId --jq '.[0].databaseId')" --exit-status
gh run list --limit 3

Mine:

STATUS  TITLE                                                   WORKFLOW  BRANCH  EVENT  ID           ELAPSED  AGE
✓       Concurrency test 2                                      CI        main    push   40222232966  1m54s    about 1 minute ago
X       Concurrency test 1                                      CI        main    push   40222230998  2s       about 1 minute ago
✓       Replace superseded runs and cap each job at 10 minutes  CI        main    push   40222229091  56s      about 2 minutes ago

The first run went to the end in 56 seconds. Test 1 was waiting behind it, and test 2 replaced it after 2 seconds, before it ran a single job; gh run view on it says cancelled, not failed. Test 2 took 1m54s: about a minute waiting for the first run, then its own run. That wait is the price of queueing on main, and it's visible right there. On a pull request, the first run would have been cancelled instead, and test 2 would have started straight away.

Jobs that never end: timeouts

What breaks. A test waits for a database that never answers, or a tool asks a question nobody can see. On GitHub's own runners, a stuck job keeps going for six hours, the default limit, before GitHub stops it. On your own Jenkins, without a limit, it can sit there forever, holding an executor that every other build is queueing for.

You added the fix in the last step: timeout-minutes: 10 on each job. The sandbox's jobs take under a minute, so ten minutes leaves plenty of room for a slow day and still turns a hang into a clear failure within ten minutes, not six hours. Pick a limit a few times longer than a normal run, not a round guess.

Jenkins has had the same thing since Lesson 2: timeout(time: 15, unit: 'MINUTES') in the Jenkinsfile's options. It's also worth wrapping a single risky step, as you'll see in my StackIT pipeline below.

Tools that change under you: pin what you install

What breaks. On Friday the pipeline is green. On Monday, a pull request that only changes a README fails at lint. Nobody touched the code: the tool released a new version over the weekend, with a new rule or a new idea of how code should be formatted, and pip install ruff fetched it. Every run on every branch now fails, and the fix has nothing to do with anyone's work.

The sandbox installed Ruff with no version since Lesson 3, and so does CivicGrid. The last run's log shows what it got:

gh run view "$(gh run list --limit 1 --json databaseId --jq '.[0].databaseId')" --log | grep -E "Successfully installed"

test UNKNOWN STEP Successfully installed iniconfig-2.3.1 packaging-26.3 pluggy-1.7.0 psycopg2-binary-2.9.13 pygments-2.21.0 pytest-9.0.3 redis-8.1.0
lint UNKNOWN STEP Successfully installed ruff-0.17.2

So pin that: Ruff goes in a file of its own, and both tools install from it. The same step turns on the cache, which the next section measures:

echo "ruff==0.17.2" > requirements-lint.txt
python3 - <<'EOF'
from pathlib import Path
p = Path(".github/workflows/ci.yml")
s = p.read_text()
setup = "      - uses: actions/setup-python@v7\n        with:\n          python-version: '3.13'\n"
def cached(req):
    return setup + f"          cache: pip\n          cache-dependency-path: {req}\n"
lint_old = setup + "      - run: pip install ruff\n"
test_old = setup + "      - run: pip install -r requirements-dev.txt\n"
assert lint_old in s and test_old in s
s = s.replace(lint_old, cached("requirements-lint.txt") + "      - run: pip install -r requirements-lint.txt\n")
s = s.replace(test_old, cached("requirements-dev.txt") + "      - run: pip install -r requirements-dev.txt\n")
p.write_text(s)
j = Path("Jenkinsfile")
t = j.read_text()
assert "pip install --quiet ruff" in t
j.write_text(t.replace("pip install --quiet ruff", "pip install --quiet -r requirements-lint.txt"))
EOF
git add requirements-lint.txt
git commit -am "Pin Ruff and cache pip downloads"
git push

Now Ruff changes when you change one line in requirements-lint.txt, in a commit of its own, where a new rule's complaints are the only thing that pull request is about.

Look at the test line above, though. You pinned pytest, psycopg2-binary and redis, and pip still chose iniconfig, packaging, pluggy and pygments itself, because they're dependencies of your dependencies. For a sandbox that's fine. For an app, the full answer is a lock file: a list of every package and its exact version, made by a tool like pip-tools or uv, so that every install is identical. Pinning the GitHub actions themselves, and keeping all these pins up to date without doing it by hand, is a security story as much as a reliability one, so it's in Lesson 7.

Caching, measured

What breaks is nothing, at first: caching is about speed. But a cache is easy to add on faith, and it's worth knowing whether yours helps. cache: pip saves pip's downloads at the end of a job and restores them at the start of the next, so packages come from GitHub's cache rather than the internet. The key is made from the requirements file named in cache-dependency-path, so changing a pin starts a fresh cache.

Here are my job times, before the cache and on the two runs after:

It didn't help. Lint installs one package and test installs seven, and downloading them is about as fast as restoring them, so the cache only adds a step. Small differences between runs are noise: waiting for a runner or for Postgres to start moves the numbers more than pip ever does. Caching pays off when installs are big, like a framework with dozens of dependencies, a large node_modules folder, or Docker layers, which is why Lesson 4's cache-from: type=gha is worth keeping.

The first run also showed something worth knowing. Lint found no cache and saved one. The test job, starting a little later, said this:

test Run actions/setup-python@v7 Cache hit for restore-key: setup-python-Linux-x64-24.04-Ubuntu-python-3.13.16-pip-ed8c5654b2f1ca54e9e86c85f5dbaa559f3dceeaa36958a85079787d0e161a32
test Run actions/setup-python@v7 Cache restored from key: setup-python-Linux-x64-24.04-Ubuntu-python-3.13.16-pip-ed8c5654b2f1ca54e9e86c85f5dbaa559f3dceeaa36958a85079787d0e161a32
test Post Run actions/setup-python@v7 Cache saved with the key: setup-python-Linux-x64-24.04-Ubuntu-python-3.13.16-pip-d1ca012a2f4a376dc25c1de7926336a5594dd9ac84b2bbf2e30685fbd5a80386

When there's no exact match, setup-python falls back to any cache whose key starts the same way, a restore key. Lint's cache, ...-pip-ed8c..., matched the start, so the test job downloaded 11 MB of Ruff it had no use for, installed its own packages anyway, then saved its own cache, ...-pip-d1ca.... Harmless here, but it's the usual way caches go quietly wrong: a near match restores something stale or unrelated, and it works well enough that nobody looks. Measure a cache before and after, and read its log once.

In Jenkins: a server that cleans up after itself

GitHub throws its runners away after every job. Jenkins doesn't: it's one machine that keeps everything, so its problems pile up. What breaks is usually the disk. Every build keeps its log and records forever, and every workspace stays where it was left, virtual environments and all. When free space runs low, Jenkins takes the machine offline, and every pipeline stops at once.

Four lines deal with most of it:

python3 - <<'EOF'
from pathlib import Path
p = Path("Jenkinsfile")
s = p.read_text()
opts = "    timeout(time: 15, unit: 'MINUTES')\n"
env = "    PIP_NO_CACHE_DIR = '1'\n"
post = "    always {\n      sh 'docker logout ghcr.io || true'\n    }\n"
assert s.count(opts) == 1 and s.count(env) == 1 and s.count(post) == 1
s = s.replace(opts, opts + "    buildDiscarder(logRotator(numToKeepStr: '10'))\n    disableConcurrentBuilds(abortPrevious: true)\n")
s = s.replace(env, env + "    CI = 'true'\n")
s = s.replace(post, post + "    cleanup {\n      deleteDir()\n    }\n")
p.write_text(s)
EOF
git commit -am "Jenkins: keep 10 builds, stop superseded builds, set CI, clean the workspace"
git push

• buildDiscarder(logRotator(numToKeepStr: '10')) keeps the last ten builds and deletes older ones, logs and all.
• disableConcurrentBuilds(abortPrevious: true) is Jenkins's concurrency group: one build at a time, and a new one stops the one still running.
• CI = 'true': Jenkins doesn't set it the way GitHub does. Without it, the new conftest.py check would never run in Jenkins.
• post { cleanup { deleteDir() } } empties the workspace after every build, whatever the result. cleanup runs after every other post block, so the logout still happens first.

Build Now once. That first build reads the new Jenkinsfile and switches the options on, and it passed, with 3 passed from tests that now see CI. Then the abort: click Build Now, wait until the build is running, and click it again. If you click twice in a few seconds, Jenkins merges the clicks into one build during its short "quiet period", and there's nothing to stop. I clicked three times, and read the logs of the stopped builds straight from Jenkins's files:

for n in 9 10; do echo "== build $n"; docker exec jenkins sh -c "sed 's#ha:////[A-Za-z0-9+/=]*##g' /var/jenkins_home/jobs/sandbox/builds/$n/log" | grep -E -i "superseded|interrupt|\(Checks\)|Finished"; done

The sed removes Jenkins's web-page formatting from the raw log file. Mine:

== build 9
[Pipeline] { (Checks)
Superseded by #10
Finished: NOT_BUILT
== build 10
[Pipeline] { (Checks)
Superseded by #11
Sending interrupt signal to process
Finished: NOT_BUILT

Each new build stopped the one before during its checks, and only #11 went on to build and push. Jenkins marks a stopped build NOT_BUILT, not failed, so it doesn't look like broken code.

The cleanup showed up in two places. Build #8, the first after a cleaned-up build, began with Cloning the remote Git repository instead of the usual Fetching changes, because its workspace was empty. That's the trade-off: a clean disk, and a fresh download of the code every time. And the workspace folder itself was gone:

docker exec jenkins sh -c 'du -sh /var/jenkins_home/jobs/sandbox/builds /var/jenkins_home/workspace/sandbox* 2>/dev/null'

1.4M /var/jenkins_home/jobs/sandbox/builds
4.0K /var/jenkins_home/workspace/sandbox@tmp

Seven builds of records take 1.4 MB, which is nothing yet; buildDiscarder only starts deleting after the tenth. On a real server, with long logs and saved files from hundreds of builds, that number is what grows. The setting is cheap insurance, not a rescue.

A long-running Jenkins collects other leftovers too, like containers from builds that were stopped partway and old images on its Docker engine. Lesson 9, on running Jenkins yourself, deals with those.

options {
    timeout(time: 45, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '10'))
}

Spot the problem

The friendly default. A team's tests skip when the database isn't configured, "so they work on anyone's laptop". The pipeline has been green for months. Is it testing the database code? There's no way to tell from the green tick. Look for skipped in the test output, or better, make CI fail when those settings are missing, so a skip can only ever happen on a laptop.

The deploy that cancelled itself. A workflow builds and deploys on every push to main, with cancel-in-progress: true. Two people merge within a minute of each other. What can go wrong? The second push cancels the first run partway through its deploy, which can leave a half-finished release, with some parts updated and some not. Deploys should queue (cancel-in-progress: false), and only checks on pull requests should cancel.

The Monday failure. On Monday, every open pull request fails at lint, though none of them changed any code since Friday. What's the first thing to check? Which tool versions the failing run installed, compared with Friday's. An unpinned tool that released over the weekend is the usual cause, and pinning it turns that surprise into a pull request you choose to merge.

Where you are now

Your pipelines now cope with time as well as code. A missing database setting fails the run instead of skipping tests, pushes queue or cancel instead of piling up, a hung job fails in ten minutes, and Ruff changes only when you change it. You've measured a cache and found that, here, it doesn't help, and seen one match the wrong thing. And Jenkins now keeps ten builds, stops superseded ones and empties its workspace after every build. Lesson 7 is security scanning: what's inside your image and your dependencies, and how to pin and update the actions and tools your pipeline trusts.

Check yourself

Key terms

• conftest.py: a file pytest loads automatically, used here to stop a run that would skip its integration tests.
• CI variable: set to true by GitHub on every runner; Jenkins needs it set by hand.
• Concurrency group: a named queue that lets one run go at a time, with at most one waiting.
• cancel-in-progress: whether a newer push also stops the run that's already going.
• timeout-minutes: the most a GitHub job may run before it's stopped.
• Pinning: naming an exact version, like ruff==0.17.2, so an install is the same every time.
• Lock file: a list of every package, dependencies included, at exact versions.
• Cache key: the name a cache is saved under, made from your requirements file.
• Restore key: a fallback that restores a cache whose key only starts the same way.
• buildDiscarder: the Jenkins option that deletes old builds beyond a number you set.
• disableConcurrentBuilds(abortPrevious: true): one Jenkins build at a time, with a new one stopping the old.
• NOT_BUILT: the result Jenkins gives a build that a newer one replaced.
• deleteDir(): the built-in step that empties the current workspace.

Go deeper

• GitHub's Control the concurrency of workflows and jobs (opens in a new tab), including the one-running, one-waiting rule.
• The setup-python guide to caching packages (opens in a new tab), with every option for cache-dependency-path.
• Jenkins's Pipeline syntax: options (opens in a new tab), the full list of what can go in options { }.