Running Jenkins yourself

By the end you'll know

Why Jenkins didn't build when you pushed, and how polling and webhooks fix it
How a build can clean up after itself without breaking the build after it
Why a restart policy decides whether Jenkins is there in the morning
How to back up jenkins_home, and prove the backup works by restoring it
How to check whether your Jenkins is current, and why plugins are the hard part
Why builds shouldn't run on the server that holds the keys

Where you are

Since Lesson 2, your Jenkins has run every stage your GitHub workflow runs: lint, tests, audit, build, scan, push and an approved deploy. In all that time you've treated it as a tool. This lesson treats it as what it also is: a server you own. Everything GitHub does behind the scenes for Actions, someone has to do for Jenkins, and that someone is you.

There's no new setup in this lesson. Everything happens on the Jenkins you already have.

Renting and owning

Think of renting a flat versus owning a house. In a flat, a leaking roof is a phone call: the landlord sends someone, and you never think about the boiler. In a house you own, you can knock down walls and build what you like, but the roof, the boiler and the gutters are yours, and nobody reminds you about them. GitHub-hosted runners are the flat. A Jenkins you run is the house: more freedom, and a list of chores.

10 of 10 chores are yours.

Patch the machine that runs buildsYouGitHub replaces its runners for every job. Your Jenkins host needs updates like any server.
Upgrade the CI tool and its pluginsYouYour Jenkins has 95 plugins. Each one is code that needs updating, and a backup first.
Back up jobs, history and settingsYouGitHub keeps run history for you. jenkins_home is one volume, and losing it loses everything.
Clear disk space after buildsYouGitHub throws each runner away. Jenkins keeps workspaces, images and caches until something removes them.
Start a build when code is pushedYouActions sees every push. Jenkins needs a webhook it can receive, or polling.
Keep it running after a restartYouYour containers stayed off after a restart until the policy said unless-stopped.
Protect the key that decrypts secretsYouGitHub holds your secrets’ encryption keys. On Jenkins, master.key sits in jenkins_home, and in every backup of it.
Keep builds away from that keyYouEach GitHub job gets a fresh machine. Jenkins runs builds on the controller unless you set up separate agents.
Pin actions, tools and imagesYouLessons 6 and 7. Neither tool does this for you.
Give each job only the access it needsYouLesson 5. permissions: on GitHub, credential scoping on Jenkins.

The rest of this lesson works through the list, starting with the chores that broke during this track.

"I pushed and nothing happened"

What breaks. Every Jenkins build in this track started when you clicked Build Now. GitHub Actions sees every push because it lives where the code lives. Jenkins only builds when something tells it to, and nothing was.

There are two ways to tell it:

A webhook: GitHub sends a message to Jenkins on every push. It's instant, and it's what a real server uses. But GitHub has to be able to reach Jenkins, and yours is on localhost, which nothing on the internet can reach.
Polling: Jenkins asks GitHub every few minutes whether anything changed. It's slower and makes more requests, but it works from anywhere.

Use polling here. The same change also limits deploys to builds a person started, since a polled build would otherwise sit at the approval question for 15 minutes and fail, and adds cleanup that the next section explains. On your laptop, in ~/Apps/cicd-sandbox:

BASH

cd ~/Apps/cicd-sandbox
python3 - <<'EOF'
from pathlib import Path
p = Path("Jenkinsfile")
s = p.read_text()
reps = [
 ("  environment {\n", "  triggers {\n    pollSCM('H/2 * * * *')\n  }\n  environment {\n"),
 ("withRun('-e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=visits_test')", 'withRun("--label sandbox-build=${env.BUILD_TAG} -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=visits_test")'),
 ("docker.image('redis:7.2.7-alpine').withRun()", 'docker.image(\'redis:7.2.7-alpine\').withRun("--label sandbox-build=${env.BUILD_TAG}")'),
 ("        expression { env.GIT_BRANCH == 'origin/main' }\n", "        allOf {\n          expression { env.GIT_BRANCH == 'origin/main' }\n          triggeredBy 'UserIdCause'\n        }\n"),
 ('          docker run --rm "$REF" "  Jenkins Production "\n', '          docker run --rm "$REF" "  Jenkins Production "\n          docker image rm "$REF"\n'),
 ("      sh 'docker logout ghcr.io || true'\n", "      sh 'docker logout ghcr.io || true'\n      sh 'docker ps -aq --filter \"label=sandbox-build=$BUILD_TAG\" | xargs -r docker rm -f -v'\n      sh 'docker image rm cicd-sandbox:$IMAGE_TAG ghcr.io/kwamib/cicd-sandbox-jenkins:$IMAGE_TAG 2>/dev/null || true'\n"),
]
for old, new in reps:
    assert s.count(old) == 1, old
    s = s.replace(old, new)
p.write_text(s)
EOF
git commit -am "Jenkins: poll for changes, deploy only on manual builds, clean up after each build"
git push

Use your own username in place of kwamib, as before.

pollSCM('H/2 * * * *') checks for new commits about every two minutes. It's cron syntax, and H means "a fixed minute picked for this job", so a server with many jobs doesn't have them all polling at the same second.
triggeredBy 'UserIdCause' runs Deploy only when a person started the build. Polled builds still check, scan and push.

Then push an empty commit and wait, without touching Jenkins:

BASH

git commit --allow-empty -m "Polling test: no click needed"
git push
sleep 240
n=$(docker exec jenkins sh -c 'ls /var/jenkins_home/jobs/sandbox/builds | grep "^[0-9]" | sort -n | tail -1')
docker exec jenkins sh -c "sed 's#ha:////[A-Za-z0-9+/=]*##g' /var/jenkins_home/jobs/sandbox/builds/$n/log" | grep -E -i "Started by|Commit message|\(Deploy\)|skipped|Finished"

Mine, four minutes later:

OUTPUT

Started by user Kwami
Commit message: "Jenkins: deploy from GHCR by digest, not the local name"
[Pipeline] { (Deploy)
Finished: SUCCESS

That's the last build from Lesson 8. No new build happened. Here's the catch: a triggers { } block lives in the Jenkinsfile, but Jenkins only reads the Jenkinsfile when it runs it. Until the new file has run once, the job still has its old settings, and the old settings said nothing about polling.

So Build Now once (it will stop at Deploy, since you started it; click Deploy), then push one more empty commit and check the last two builds:

BASH

git commit --allow-empty -m "Polling test 3: after the trigger is registered"
git push
sleep 240
for n in $(docker exec jenkins sh -c 'ls /var/jenkins_home/jobs/sandbox/builds | grep "^[0-9]" | sort -n | tail -2'); do echo "== build $n"; docker exec jenkins sh -c "sed 's#ha:////[A-Za-z0-9+/=]*##g' /var/jenkins_home/jobs/sandbox/builds/$n/log" | grep -E -i "Started by|Commit message|\(Deploy\)|skipped|Finished"; done

OUTPUT

== build 24
Started by user Kwami
Commit message: "Polling test 2: Jenkins should find this by itself"
[Pipeline] { (Deploy)
Finished: SUCCESS
== build 25
Started by an SCM change
Commit message: "Polling test 3: after the trigger is registered"
[Pipeline] { (Deploy)
Stage "Deploy" skipped due to when conditional
Finished: SUCCESS

Build #24 was the click that registered the trigger. (Its commit says "Polling test 2" because I'd pushed a second test while waiting; yours will name your first test commit.) Build #25 started by itself, Started by an SCM change, and its Deploy stage was skipped because nobody asked for it.

Builds that leave things behind

What breaks. GitHub throws away the runner after every job, so whatever a job leaves behind disappears with it. Jenkins keeps everything. In Lesson 6, a build stopped by abortPrevious left its Redis sidecar running, still there 17 minutes later:

OUTPUT

CONTAINER ID   IMAGE                COMMAND                  CREATED          STATUS          PORTS      NAMES
015fad11a4ab   redis:7.2.7-alpine   "docker-entrypoint.s…"   17 minutes ago   Up 17 minutes   6379/tcp   competent_volhard

The cleanup inside withRun never ran, because the build was interrupted first. Anything a build starts has to be cleaned up by something that still runs after an interruption, and that's post.

There's a trap in how you write it, and I fell into it. My first fix gave every sidecar the same label, sandbox-sidecar, and had post remove every container with that label. Then a stopped build, #15, was still shutting down when its replacement, #16, had started its own databases:

OUTPUT

== build 15
Superseded by #16
+ docker ps -aq --filter label=sandbox-sidecar
+ xargs -r docker rm -f -v
Finished: NOT_BUILT
== build 16
+ docker stop 31695d5cc6d03ff7edb590cccf6a36d5eb57d66a1baec4fce48b48f19f77de63
Error response from daemon: No such container: 31695d5cc6d03ff7edb590cccf6a36d5eb57d66a1baec4fce48b48f19f77de63
Failed in branch Tests
Finished: FAILURE

Build #15's cleanup removed #16's Postgres and Redis. abortPrevious doesn't wait for the old build to finish its post before starting the new one, so for a few seconds they overlap. A cleanup must only touch what its own build created. That's why the change above labels each sidecar with BUILD_TAG, which Jenkins sets to something unique like jenkins-sandbox-24, and sweeps only that label. From build #24's log:

OUTPUT

+ docker run -d --label sandbox-build=jenkins-sandbox-24 -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=visits_test postgres:17
+ docker run -d --label sandbox-build=jenkins-sandbox-24 redis:7.2.7-alpine
+ docker image rm ghcr.io/kwamib/cicd-sandbox-jenkins@sha256:67468e0a962a455621ce7624c114d99e7fb5ccf036e99871d131d8a1327801ea
Untagged: ghcr.io/kwamib/cicd-sandbox-jenkins@sha256:67468e0a962a455621ce7624c114d99e7fb5ccf036e99871d131d8a1327801ea
+ docker ps -aq --filter label=sandbox-build=jenkins-sandbox-24
+ xargs -r docker rm -f -v

The build also removes its own images afterwards: the one Deploy pulled, and the local copies in post. Each one is already safe in GHCR. Check what's left on Jenkins's Docker engine:

BASH

docker exec jenkins docker ps -a
docker exec jenkins docker image ls
docker exec jenkins docker system df -v | sed -n '/Local Volumes/,/Build cache/p'

OUTPUT

CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
IMAGE                                                                                   ID             DISK USAGE   CONTENT SIZE   EXTRA
aquasec/trivy@sha256:af6acf9a6b85dfe389a1941505c0ce9efef52a4719635e1a962f022a3d855daa   af6acf9a6b85        243MB         58.6MB
postgres:17                                                                             b8faeff77126        667MB          166MB
python:3.13-slim                                                                        d98cc29d3266        204MB         45.5MB
redis:7.2.7-alpine                                                                      ddd16a9b1575       61.2MB         17.8MB
Local Volumes space usage:

VOLUME NAME   LINKS     SIZE
trivy-cache   0         1.638GB

No leftover containers, and only the images the pipeline needs every time. Then there's the surprise: Trivy's cache is 1.6 GB, more than all the images together. A cache that speeds things up can quietly become the biggest thing on the disk. It's fine to keep, but it's the first thing to look at when space runs low, and safe to delete, since Trivy downloads it again.

A server that's there in the morning

What breaks. Your Mac restarts for an update, or Docker Desktop restarts. Does Jenkins come back?

BASH

docker inspect jenkins jenkins-docker --format '{{.Name}} restart: {{.HostConfig.RestartPolicy.Name}}'
docker update --restart unless-stopped jenkins jenkins-docker
docker inspect jenkins jenkins-docker --format '{{.Name}} restart: {{.HostConfig.RestartPolicy.Name}}'

OUTPUT

/jenkins restart: on-failure
/jenkins-docker restart: on-failure
jenkins
jenkins-docker
/jenkins restart: unless-stopped
/jenkins-docker restart: unless-stopped

on-failure restarts a container that crashes, but not one that was running when Docker itself stopped, so after a restart Jenkins stayed off until someone noticed. unless-stopped brings both back, unless you stopped them on purpose. On a real server, this is the systemd service, or the cloud's auto-recovery, and it's the difference between "Jenkins is down" being a page at 3am or not happening at all.

Losing jenkins_home

What breaks. One volume, jenkins_home, holds every job, every build record, every user and every stored credential. Lose the volume, through a deleted container with -v, a failed disk, or a mistaken cleanup, and Jenkins is new again: no jobs, no history, no GHCR token.

Look at what's in it:

BASH

docker exec jenkins sh -c 'du -sh /var/jenkins_home; du -sh /var/jenkins_home/* 2>/dev/null | sort -h | tail -5'
docker exec jenkins sh -c 'ls /var/jenkins_home/secrets/'

OUTPUT

257M /var/jenkins_home
1.4M /var/jenkins_home/caches
2.5M /var/jenkins_home/jobs
3.5M /var/jenkins_home/updates
66M /var/jenkins_home/war
183M /var/jenkins_home/plugins
hudson.console.ConsoleNote.MAC
hudson.model.Job.serverCookie
hudson.model.User.DIRNAMES
hudson.util.Secret
jenkins.model.Jenkins.crumbSalt
jenkins.security.csp.ReportingContext.key
master.key
org.jenkinsci.main.modules.instance_identity.InstanceIdentity.KEY
org.jenkinsci.plugins.workflow.log.ConsoleAnnotators.consoleAnnotator

Your jobs are 2.5 MB. The plugins are 183 MB. And in secrets/ are master.key and hudson.util.Secret, the keys Jenkins uses to decrypt every stored credential. A backup of jenkins_home includes them, so the backup can decrypt your GHCR token. Treat it as a secret: a folder only you can read, outside every git repository, never in a shared drive unencrypted.

Back it up

BASH

mkdir -p ~/Backups/jenkins
chmod 700 ~/Backups/jenkins
docker run --rm -v jenkins_home:/var/jenkins_home:ro -v ~/Backups/jenkins:/backup alpine:3.22 sh -c 'tar czf /backup/jenkins_home-$(date +%Y%m%d-%H%M).tgz -C /var/jenkins_home --exclude=./war --exclude=./caches --exclude=./workspace . && chmod 600 /backup/*.tgz'
ls -la ~/Backups/jenkins

A throwaway Alpine container reads the volume, mounted read-only with :ro, and writes a compressed copy to your folder. It skips war, caches and workspace, which Jenkins rebuilds by itself. Mine:

OUTPUT

-rw-------@ 1 mayowababatola  staff  148405490 Oct 25 22:16 jenkins_home-20261026-0216.tgz

148 MB, readable only by me. Notice the time in the name, 0216, against 22:16 on the file: the container's clock is in UTC, four hours ahead of mine. Whatever you choose, say which it is, or a 2am restore will start with arithmetic.

Prove it works

A backup you've never restored is a hope, not a backup. Restore it into a second Jenkins, with no network, so it can't poll GitHub and start building:

BASH

docker volume create jenkins_home_restore
docker run --rm -v jenkins_home_restore:/var/jenkins_home -v ~/Backups/jenkins:/backup:ro alpine:3.22 sh -c 'tar xzf "/backup/$(ls /backup | tail -1)" -C /var/jenkins_home && chown -R 1000:1000 /var/jenkins_home'
docker run -d --name jenkins-restore --network none -v jenkins_home_restore:/var/jenkins_home jenkins-with-docker
sleep 60
docker logs jenkins-restore 2>&1 | grep -E "fully up and running" | head -1

OUTPUT

2026-10-26 03:38:02.609+0000 [id=53] INFO hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running

"Fully up" isn't proof on its own: Jenkins starts happily on an empty folder too. Log in to the copy and ask it for its jobs and credentials. Paste the two read lines one at a time; the password won't show:

BASH

read "JUSER?Jenkins username: "

BASH

read -s "JPW?Jenkins password: "

BASH

echo
docker exec -e JUSER="$JUSER" -e JPW="$JPW" jenkins-restore sh -c 'curl -s -o /dev/null -w "restored Jenkins login: %{http_code}\n" -u "$JUSER:$JPW" http://localhost:8080/api/json; curl -s -w "\n(credentials: %{http_code})\n" -u "$JUSER:$JPW" "http://localhost:8080/credentials/store/system/domain/_/api/json?tree=credentials\[id,typeName\]"; curl -s -w "\n(jobs: %{http_code})\n" -u "$JUSER:$JPW" "http://localhost:8080/api/json?tree=jobs\[name\]"'
unset JPW

OUTPUT

restored Jenkins login: 200
{"_class":"com.cloudbees.plugins.credentials.CredentialsStoreAction$DomainWrapper","credentials":[{"id":"ghcr","typeName":"Username with password"}]}
(credentials: 200)
{"_class":"hudson.model.Hudson","jobs":[{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowJob","name":"first-pipeline"},{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowJob","name":"sandbox"}]}
(jobs: 200)

Your user, the ghcr credential and both jobs came back. Then remove the copy:

BASH

docker rm -f jenkins-restore
docker volume rm jenkins_home_restore

read "NAME?prompt" is how zsh, the Mac's shell, shows a prompt; plain read waits silently, which looks like a frozen terminal. If you're not sure of your Jenkins username, docker exec jenkins sh -c 'for f in /var/jenkins_home/users/*/config.xml; do grep -m1 "<id>" "$f"; done' lists them. You'll also see users nobody created, one per commit author Jenkins has seen, like your GitHub address and the deploy bot's. They're records, not accounts anyone can log in with.

Staying current

What breaks. Jenkins and its plugins get security fixes all the time, and a server nobody upgrades slowly becomes a server with known holes, reachable by whoever can reach it. Check where you are:

BASH

docker exec jenkins sh -c 'echo "Jenkins $JENKINS_VERSION"; java -version 2>&1 | head -1'
docker exec jenkins sh -c 'ls /var/jenkins_home/plugins/*.jpi 2>/dev/null | wc -l'
curl -sL https://updates.jenkins.io/stable/latestCore.txt; echo

OUTPUT

Jenkins 2.580.1
openjdk version "21.0.12.1" 2026-08-18 LTS
95
2.580.1

The .1 on the end marks an LTS (long-term support) release: one picked every few months and then only given fixes, which is the line to run a real server on. Mine matched the newest LTS, so there was nothing to do today. That's the outcome you want most weeks, and the check takes one command.

The core is the easy part. The other 95 entries are plugins, each one code from a different project, updated on its own schedule, and the usual reason a Jenkins upgrade goes wrong. The habit that makes upgrades safe is the one you just practised: take a backup, upgrade, and if anything is wrong, restore. Manage Jenkins, Plugins, Updates shows what's waiting.

Builds next to the keys

Every Jenkins log in this track started the same way:

OUTPUT

Running on Jenkins
 in /var/jenkins_home/workspace/sandbox

"Jenkins" there is the controller, the server itself. Your builds ran on it, in the same container as master.key:

BASH

docker exec jenkins grep -m1 numExecutors /var/jenkins_home/config.xml

OUTPUT

  <numExecutors>2</numExecutors>

Two builds at a time can run on the controller. Any step in any build there, including a malicious one from a compromised dependency, runs with access to jenkins_home: the keys, every credential and every other job's settings. That's why Jenkins's own advice is zero executors on the controller: builds run on separate machines called agents, which get only what each build needs, and the controller just schedules work and stores results. GitHub works this way by design: every job gets a fresh machine that never sees the keys to anything else.

Setting up an agent is its own project, so this lesson stops at explaining it. On your laptop, the controller doing the building is fine. On a server other people use, it's the first thing to change.

In my real setup

My earlier Jenkins on AWS ran on an EC2 instance behind a load balancer called jenkins-alb. When I stopped using it, I stopped the instance, and that felt like enough. It wasn't: the load balancer and its public IP address kept charging every month with no traffic at all. I found it months later, tracing an AWS bill through Cost Explorer: jenkins-alb had been running since May 2025, in front of a stopped Jenkins. I archived the instance as an image (an AMI, Amazon Machine Image), so it can come back, then deleted the load balancer and terminated the instance. Owning Jenkins includes owning everything around it, and paying for it until you tear it down.

That's a big part of why CivicGrid's pipeline runs on GitHub Actions. There's no server to patch, back up or forget, and the chores this lesson covers are GitHub's. I'd still choose Jenkins where a team needs builds inside its own network, unusual hardware, or full control over where code and secrets go. That's a real reason, and many companies have it. "We always have" isn't one.

Spot the problem

The cleanup that broke the next build. A Jenkinsfile's post step removes every container with the label ci-sidecar. Builds use abortPrevious. Now and then, a build fails because its database disappeared mid-test. Why? A stopped build's post runs while its replacement is already running, and removes the replacement's containers too. Label each build's containers with its own BUILD_TAG, and sweep only that.

The backup nobody tested. A team backs up jenkins_home every night with a script. After a disk failure, they find the backups are all a few kilobytes: the script had been archiving an empty folder since a path changed. What would have caught it? Restoring one, now and then, and checking it has the jobs and credentials. A backup is only as good as its last restore.

The trigger that never fired. Someone adds triggers { pollSCM('H/5 * * * *') } to a Jenkinsfile, pushes, and waits. Nothing builds, all afternoon. What's missing? One run of the new Jenkinsfile. Jenkins registers triggers when it runs the file, so the first build after adding one has to be started some other way.

Where you are now

That's the whole track. You've built the same pipeline twice, in GitHub Actions and in Jenkins: lint, tests against real services, an audit and an image scan, a multi-processor image pushed by digest, and a deploy that waits for a person and can be undone with git revert. And now you know the second job that comes with Jenkins: it starts on a push, cleans up after itself without touching other builds, comes back after a restart, has a backup that's been restored, is current, and you know why its builds belong on agents. Whichever tool your next team uses, you'll know what it's doing for you and what it isn't.

Check yourself

Why didn't Jenkins build after the push that added pollSCM?
Why does the sweep use the label sandbox-build=$BUILD_TAG?
What did changing the restart policy to unless-stopped fix?
Why must a backup of jenkins_home be kept private?
The restored Jenkins said fully up and running. Why check more?
Why does Jenkins advise zero executors on the controller?

Key terms

Controller: the Jenkins server itself, which schedules builds and stores jobs, history and credentials.
Agent: a separate machine that runs builds for the controller.
Executor: a slot for one build at a time; numExecutors sets how many a machine has.
Webhook: a message GitHub sends to another server when something happens, like a push.
Polling (pollSCM): Jenkins checking the repository on a schedule for new commits.
BUILD_TAG: a name Jenkins sets for each build, like jenkins-sandbox-24, unique across builds.
Restart policy: Docker's rule for when to restart a container; unless-stopped survives restarts.
jenkins_home: the folder, here a volume, that holds everything Jenkins knows.
master.key: the key Jenkins uses, with hudson.util.Secret, to decrypt stored credentials.
Restore test: proving a backup works by restoring it somewhere and checking its contents.
LTS (long-term support): a release line picked for stability and given only fixes.

Go deeper

Jenkins's Backing-up/Restoring Jenkins (opens in a new tab), including what to leave out.
Jenkins's Controller isolation (opens in a new tab), on why builds belong on agents.
Jenkins's LTS release line (opens in a new tab), and how often it moves.