Resources CI/CD CI/CD with GitHub Actions and Jenkins

CI/CD with GitHub Actions and Jenkins · Lesson 8 of 9 · about 25 minutes

From image to production

The last step of the pipeline: deploy the exact image that was tested, by its digest, through an approval gate, as a commit that a Kubernetes cluster pulls. Building for more than one kind of processor, rolling back with git revert, and the same gate in Jenkins.

By the end you'll know

  • Why production should run the exact digest CI tested, never latest or a rebuild
  • Why an image built for one kind of processor can't run on another, and how to build for both
  • How a GitHub Environment holds a deploy until someone approves it
  • How GitOps turns a deploy into a commit, and a rollback into git revert
  • How Jenkins asks for approval with input, and deploys by digest from the registry

A heads-up: this is the longest lesson in the track. It's the one that ties the others together, and there's a real cluster at the end of it. Take it in two sittings if you like; the break after "The cluster pulls" is a natural one.

Where you are

Your pipelines lint, test, audit, scan, and push an image. Nothing yet takes that image anywhere. This lesson adds the last step, delivery: getting the tested image into production, on purpose, with a person deciding when, and a way back.

"Production" here is a Kubernetes cluster on your laptop, made with minikube. Starting it is the lesson's one setup step. If you don't have minikube yet, its Get Started page takes a few minutes.

The dish that was tasted

Think of a restaurant kitchen. The head chef tastes a dish before it goes out. If the waiter then carries a different plate, or the cook makes the dish again "the same way" because the first one went cold, the tasting meant nothing. The customer gets something nobody checked. A deploy has the same rule: the thing that goes out has to be the thing that was checked, not something like it.

Two common deploys break that rule without anyone noticing:

  • Deploying latest. You approve your run, and by the time the deploy pulls latest, a teammate's merge has moved it. Lesson 4's tag timeline showed how latest drifts.
  • Rebuilding for production. The same commit, built again on deploy day, pulls a newer base image and newer packages. It has a different digest, and no test or scan ever saw it.

The fix is to build once, and promote the digest: the build job reports the digest of the image it pushed, and the deploy uses exactly that.

Try it: how does production find its image?

Tested in CI

30d2d78 sha256:b13eb29d…

built, tested and scanned by CI

Running in production

9f41c0a sha256:5c0e77a1…

a teammate’s merge, pushed while you waited

Not what was tested

Between your run finishing and the deploy, someone else merged and their run moved latest. Production pulled their image, which your approval never covered.

A production to deploy to

Start the cluster, and check kubectl is talking to it. On your laptop:

BASH
minikube start
kubectl config current-context
kubectl get nodes

current-context must say minikube. If you use kubectl for anything else, like a cloud cluster, minikube start has just switched you away from it, so check the context every time before you change anything.

The sandbox app is a command that runs and exits, not a web server, so in Kubernetes it's a CronJob: a job that runs on a schedule, here once an hour. You'll also start it by hand to see it run. A plain Deployment expects its program to keep running, and would restart it forever.

Write the production file, pointing at the image that's live today. Find its digest first:

BASH
cd ~/Apps/cicd-sandbox
docker buildx imagetools inspect ghcr.io/kwamib/cicd-sandbox:latest | grep -m1 Digest
OUTPUT
Digest:    sha256:bc935f61f9792fe48be937bde4e64215234fb5e768b4eb403760866571d16e29

Use your own username in lowercase in place of kwamib, and your own digest in the file:

BASH
mkdir -p deploy
cat > deploy/app.yaml <<'EOF'
apiVersion: batch/v1
kind: CronJob
metadata:
  name: visits
spec:
  schedule: "0 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: visits
              image: ghcr.io/kwamib/cicd-sandbox@sha256:bc935f61f9792fe48be937bde4e64215234fb5e768b4eb403760866571d16e29
              args: ["  Production Run "]
EOF
kubectl apply -f deploy/app.yaml
kubectl create job --from=cronjob/visits visits-manual-1
sleep 20
kubectl get pods -l job-name=visits-manual-1

The image: line names the image by digest, @sha256:..., not by tag. kubectl create job --from=cronjob/visits runs the CronJob now instead of waiting for the hour. Mine:

OUTPUT
NAME                    READY   STATUS             RESTARTS   AGE
visits-manual-1-qw9qt   0/1     ImagePullBackOff   0          20s

Built for the wrong processor

ImagePullBackOff means the cluster couldn't get the image. Ask it why:

BASH
kubectl describe pod -l job-name=visits-manual-1 | grep -E "Failed|no match" | head -3
docker buildx imagetools inspect ghcr.io/kwamib/cicd-sandbox:latest | grep Platform
OUTPUT
  Warning  Failed     18s (x5 over 95s)  kubelet            Error: ImagePullBackOff
  Warning  Failed     6s (x4 over 96s)   kubelet            Failed to pull image "ghcr.io/kwamib/cicd-sandbox@sha256:bc935f61f9792fe48be937bde4e64215234fb5e768b4eb403760866571d16e29": no matching manifest for linux/arm64/v8 in the manifest list entries
  Warning  Failed     6s (x4 over 96s)   kubelet            Error: ErrImagePull
  Platform:    linux/amd64
  Platform:    unknown/unknown

GitHub's runners use Intel-style processors, amd64, so that's all the image was built for. My laptop, like every recent Mac, uses ARM, arm64, and so does the cluster running on it. (unknown/unknown is the build record Docker attaches to each image, not a second image.) Every gate passed, and production still couldn't run it, because no gate asked. The same thing happens in the cloud: AWS's Graviton servers are ARM too. If your laptop is Intel-based, this step worked for you, but the fix below still matters for any ARM server you deploy to.

The fix is to build for both, which the next step does.

The deploy job

Three things go into the pipeline at once:

  1. Build for both processors. platforms: linux/amd64,linux/arm64 on the build step, plus docker/setup-qemu-action, which lets an amd64 runner build arm64 images by emulating the other processor. The script looks up its latest release and pins it by commit ID, as in Lesson 7.
  2. Hand the digest on. The build job gets outputs: digest, taken from the build step, so the next job knows exactly which image was pushed.
  3. A deploy job that runs only on pushes to main, waits for approval in an environment called production, then writes the digest into deploy/app.yaml and commits it. It's the only job with contents: write, which lets it push to the repository; least privilege, from Lesson 5.

That last part is GitOps: production is whatever a file in Git says, and the only way to change production is to change the file. Every deploy is a commit, with an author, a time and a reason, and going back is a revert.

The gate: a GitHub Environment

An environment is a named target, like production, with rules a job must pass before it runs. Make one that needs your approval, and only accepts deploys from main:

BASH
echo "{\"reviewers\":[{\"type\":\"User\",\"id\":$(gh api user --jq .id)}],\"deployment_branch_policy\":{\"protected_branches\":false,\"custom_branch_policies\":true}}" | gh api -X PUT repos/Kwamib/cicd-sandbox/environments/production --input - --jq '.name, (.protection_rules[] | .type)'
gh api -X POST repos/Kwamib/cicd-sandbox/environments/production/deployment-branch-policies -f name=main --jq .name

Use your own username in the two repos/ paths. gh api user --jq .id fills in your numeric GitHub user ID, which isn't a secret. Mine:

OUTPUT
production
required_reviewers
branch_policy
main

Add the job

BASH
python3 - <<'EOF'
import subprocess
from pathlib import Path
def gh(*a):
    return subprocess.run(["gh", "api", *a], capture_output=True, text=True, check=True).stdout.strip()
tag = gh("repos/docker/setup-qemu-action/releases/latest", "--jq", ".tag_name")
sha = gh(f"repos/docker/setup-qemu-action/commits/{tag}", "--jq", ".sha")
print(f"docker/setup-qemu-action@{tag} -> {sha}")
p = Path(".github/workflows/ci.yml")
s = p.read_text()
buildx = "      - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4\n"
head = "  build:\n    needs: [lint, test, audit, scan]\n    runs-on: ubuntu-24.04\n    timeout-minutes: 10\n"
bpa = "      - uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7\n        with:\n          context: .\n"
for x in (buildx, head, bpa):
    assert s.count(x) == 1, x
s = s.replace(buildx, f"      - uses: docker/setup-qemu-action@{sha} # {tag}\n" + buildx)
s = s.replace(head, head + "    outputs:\n      digest: ${{ steps.push.outputs.digest }}\n")
s = s.replace(bpa, "      - id: push\n        uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7\n        with:\n          context: .\n          platforms: linux/amd64,linux/arm64\n")
s = s.rstrip("\n") + "\n" + """
  deploy:
    needs: build
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-24.04
    timeout-minutes: 10
    environment: production
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
      - name: Point production at the image this run built
        env:
          DIGEST: ${{ needs.build.outputs.digest }}
        run: |
          sed -i -E "s#(image: ghcr.io/kwamib/cicd-sandbox)@sha256:[0-9a-f]+#\\1@${DIGEST}#" deploy/app.yaml
          git diff
          git config user.name "github-actions[bot]"
          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git commit -am "Deploy ${GITHUB_SHA::7} to production"
          git push
"""
p.write_text(s)
EOF
git add deploy/app.yaml
git commit -am "Build for amd64 and arm64, and deploy by digest through an approved production environment"
git push

Use your own username in the sed line. If your commit IDs from Lesson 7 differ from mine, the assert will tell you which line to adjust. Mine printed:

OUTPUT
docker/setup-qemu-action@v4.4.0 -> 99012661954931238ded8c8b007157a8430204e1

Waiting at the gate

A minute or two later, the checks and the build are done, and the deploy is waiting:

BASH
RUN=$(gh run list --limit 1 --json databaseId --jq '.[0].databaseId')
gh run view "$RUN"
gh api repos/Kwamib/cicd-sandbox/actions/runs/$RUN/pending_deployments --jq '.[] | "\(.environment.name) (id \(.environment.id)) waiting for: \([.reviewers[].reviewer.login] | join(", ")), can approve: \(.current_user_can_approve)"'
OUTPUT
JOBS
✓ scan in 25s (ID 121287275130)
✓ lint in 8s (ID 121287275278)
✓ audit in 15s (ID 121287275291)
✓ test in 31s (ID 121287275331)
✓ build in 48s (ID 121287412332)
* deploy (ID 121287613582)

production (id 25296249760) waiting for: Kwamib, can approve: true

build took 48 seconds, against about 22 before: emulating the second processor roughly doubles it. The next run took 31, once the arm64 layers were in the build cache. GitHub also offers ARM runners that build arm64 natively, with no emulation.

The run sits here until someone decides. You can approve on the run's page in GitHub, with Review deployments, or from the terminal, which also records a reason:

BASH
ENV_ID=$(gh api repos/Kwamib/cicd-sandbox/actions/runs/$RUN/pending_deployments --jq '.[0].environment.id')
gh api -X POST repos/Kwamib/cicd-sandbox/actions/runs/$RUN/pending_deployments -F "environment_ids[]=$ENV_ID" -f state=approved -f comment="First deploy by digest: multi-arch image" --jq '.[0].environment'
gh run watch "$RUN" --exit-status
gh run view "$RUN" --log | sed -E 's/[0-9T:.-]+Z //' | grep -E "^deploy.*(image:|Deploy |main -> main)"

The deploy job took 6 seconds. Its change, from my log:

OUTPUT
deploy Point production at the image this run built -              image: ghcr.io/kwamib/cicd-sandbox@sha256:bc935f61f9792fe48be937bde4e64215234fb5e768b4eb403760866571d16e29
deploy Point production at the image this run built +              image: ghcr.io/kwamib/cicd-sandbox@sha256:b13eb29d9b81b0813b983188824cb697dbaeb02454df834a811969520ab14835
deploy Point production at the image this run built [main 8aac391] Deploy 30d2d78 to production
deploy Point production at the image this run built    30d2d78..8aac391  main -> main

One line changed: the old digest out, the new one in, committed as Deploy 30d2d78 to production. That commit is the deploy.

The cluster pulls

In a real setup, a tool in the cluster, like Argo CD, watches the repository and applies changes by itself. Here you're the cluster: pull the commit and apply the file. Check a few things on the way:

BASH
git pull
git log --oneline -2
gh run list --limit 2
docker buildx imagetools inspect ghcr.io/kwamib/cicd-sandbox@sha256:b13eb29d9b81b0813b983188824cb697dbaeb02454df834a811969520ab14835 | grep Platform
kubectl config current-context
kubectl delete job visits-manual-1
kubectl apply -f deploy/app.yaml
kubectl create job --from=cronjob/visits visits-manual-2
sleep 20
kubectl get pods -l job-name=visits-manual-2
kubectl logs job/visits-manual-2
kubectl get pod -l job-name=visits-manual-2 -o jsonpath='{.items[0].status.containerStatuses[0].imageID}{"\n"}'

Use your own digest. Mine:

OUTPUT
8aac391 (HEAD -> main, origin/main, origin/HEAD) Deploy 30d2d78 to production
30d2d78 Build for amd64 and arm64, and deploy by digest through an approved production environment
STATUS  TITLE                                                                   WORKFLOW  BRANCH  EVENT  ID           ELAPSED  AGE
✓       Build for amd64 and arm64, and deploy by digest through an approved p…  CI        main    push   40232585553  8m18s    about 9 minutes ago
✓       Jenkins: give pip-audit a cache folder it can write to                  CI        main    push   40231265516  1m13s    about 26 minutes ago
  Platform:    linux/amd64
  Platform:    linux/arm64
  Platform:    unknown/unknown
  Platform:    unknown/unknown
minikube
job.batch "visits-manual-1" deleted
cronjob.batch/visits configured
job.batch/visits-manual-2 created
NAME                    READY   STATUS      RESTARTS   AGE
visits-manual-2-pw4f7   0/1     Completed   0          21s
'  Production Run ' is counted as 'production run'
docker-pullable://ghcr.io/kwamib/cicd-sandbox@sha256:b13eb29d9b81b0813b983188824cb697dbaeb02454df834a811969520ab14835
  • No new run for the bot's commit. A push made with GITHUB_TOKEN doesn't start workflows, which is what stops a deploy commit from triggering another deploy, forever.
  • Both processors are in the image now.
  • The pod ran exactly b13eb29d..., the digest the build job pushed and the deploy job wrote. What CI checked is what ran.

Rolling back

What breaks. You want to undo a deploy, so you revert the deploy commit and push. But deploy/app.yaml lives in the same repository as the code, so your push starts the whole pipeline, and its deploy job, once approved, writes a newly built digest straight over your rollback. (The bot's pushes don't start runs; yours do.)

There are two usual answers: keep the deploy files in a separate repository, which is what CivicGrid does, or tell the pipeline to ignore changes that only touch deploy/. Do the second. As a workflow change, it also makes a normal run, which gives you a second release to roll back from. The until line waits for the deploy to ask for approval before approving it:

BASH
python3 - <<'EOF'
from pathlib import Path
p = Path(".github/workflows/ci.yml")
s = p.read_text()
old = "on:\n  push:\n    branches: [main]\n  pull_request:\n    branches: [main]\n"
assert s.count(old) == 1
p.write_text(s.replace(old, "on:\n  push:\n    branches: [main]\n    paths-ignore: ['deploy/**']\n  pull_request:\n    branches: [main]\n    paths-ignore: ['deploy/**']\n"))
EOF
git commit -am "Don't run CI when only deploy/ changes"
git push
sleep 15
RUN=$(gh run list --limit 1 --json databaseId --jq '.[0].databaseId')
until [ -n "$(gh api repos/Kwamib/cicd-sandbox/actions/runs/$RUN/pending_deployments --jq '.[0].environment.id' 2>/dev/null)" ]; do sleep 15; done
ENV_ID=$(gh api repos/Kwamib/cicd-sandbox/actions/runs/$RUN/pending_deployments --jq '.[0].environment.id')
gh api -X POST repos/Kwamib/cicd-sandbox/actions/runs/$RUN/pending_deployments -F "environment_ids[]=$ENV_ID" -f state=approved -f comment="Second release" --jq '.[0].environment'
gh run watch "$RUN" --exit-status

Then run release 2, and roll it back:

BASH
gh api repos/Kwamib/cicd-sandbox/actions/runs/$RUN/approvals --jq '.[] | "\(.user.login) \(.state): \(.comment)"'
git pull -q
git show --stat --format='%h %an: %s' HEAD | head -1
grep "image:" deploy/app.yaml
kubectl apply -f deploy/app.yaml
kubectl create job --from=cronjob/visits visits-release-2
sleep 20
kubectl get pod -l job-name=visits-release-2 -o jsonpath='{.items[0].status.phase} {.items[0].status.containerStatuses[0].imageID}{"\n"}'
git revert --no-edit HEAD
grep "image:" deploy/app.yaml
git push
sleep 20
gh run list --limit 2
kubectl apply -f deploy/app.yaml
kubectl create job --from=cronjob/visits visits-rollback
sleep 20
kubectl get pod -l job-name=visits-rollback -o jsonpath='{.items[0].status.phase} {.items[0].status.containerStatuses[0].imageID}{"\n"}'

Mine, without the kubectl and git push progress lines:

OUTPUT
Kwamib approved: Second release
fbc0d73 github-actions[bot]: Deploy 9849501 to production
              image: ghcr.io/kwamib/cicd-sandbox@sha256:dd4ac790dd2473b6d6c55867d5a7d58731e8dd0ea8e0bfd18c4dcc4c231693c3
Succeeded docker-pullable://ghcr.io/kwamib/cicd-sandbox@sha256:dd4ac790dd2473b6d6c55867d5a7d58731e8dd0ea8e0bfd18c4dcc4c231693c3
[main e1eda77] Revert "Deploy 9849501 to production"
              image: ghcr.io/kwamib/cicd-sandbox@sha256:b13eb29d9b81b0813b983188824cb697dbaeb02454df834a811969520ab14835
STATUS  TITLE                                                                   WORKFLOW  BRANCH  EVENT  ID           ELAPSED  AGE
✓       Don't run CI when only deploy/ changes                                  CI        main    push   40233891180  1m26s    about 2 hours ago
✓       Build for amd64 and arm64, and deploy by digest through an approved p…  CI        main    push   40232585553  8m18s    about 2 hours ago
Succeeded docker-pullable://ghcr.io/kwamib/cicd-sandbox@sha256:b13eb29d9b81b0813b983188824cb697dbaeb02454df834a811969520ab14835

GitHub kept the approval, with its reason. Release 2 ran its own digest, dd4ac790.... The revert put b13eb29d... back in the file, the push started no run, since the newest run is still "Don't run CI when only deploy/ changes", and the next pod ran the old image exactly. The old image was still in the registry, unchanged, so the rollback isn't a rebuild of something similar; it's the very image that ran before.

In Jenkins: input, and a deploy from the registry

Add a Deploy stage after Push:

BASH
python3 - <<'EOF'
from pathlib import Path
p = Path("Jenkinsfile")
s = p.read_text()
end = "  }\n  post {\n"
assert s.count(end) == 1
deploy = """    stage('Deploy') {
      when {
        expression { env.GIT_BRANCH == 'origin/main' }
      }
      steps {
        input message: 'Deploy this image to production?', ok: 'Deploy'
        sh '''
          DIGEST=$(docker image inspect --format '{{index .RepoDigests 0}}' ghcr.io/kwamib/cicd-sandbox-jenkins:$IMAGE_TAG | cut -d@ -f2)
          REF=ghcr.io/kwamib/cicd-sandbox-jenkins@$DIGEST
          echo "Deploying $REF"
          docker image rm cicd-sandbox:$IMAGE_TAG ghcr.io/kwamib/cicd-sandbox-jenkins:$IMAGE_TAG
          docker pull "$REF"
          docker run --rm "$REF" "  Jenkins Production "
        '''
      }
    }
"""
p.write_text(s.replace(end, deploy + end))
EOF
git commit -am "Jenkins: deploy by digest after a manual approval"
git push
  • input pauses the build until someone clicks Deploy in Jenkins. Lesson 6's 15-minute timeout keeps counting while it waits, so an approval nobody gives ends the build instead of holding Jenkins forever.
  • when { expression { env.GIT_BRANCH == 'origin/main' } } only deploys from main. The shorter when { branch 'main' } only works in multibranch jobs; in a plain Pipeline job like sandbox it never matches, and the stage would quietly skip every time.
  • The deploy deletes the local copies and pulls the image back from GHCR, by digest, before running it. That proves the registry copy is the one that works, not something left behind on Jenkins.

Build Now. When it reaches Deploy, open the build's Console Output: at the bottom are Deploy and Abort links. Click Deploy, then read the result:

BASH
n=$(docker exec jenkins sh -c 'ls /var/jenkins_home/jobs/sandbox/builds | grep "^[0-9]" | sort -n | tail -1')
docker exec jenkins sh -c "sed 's#ha:////[A-Za-z0-9+/=]*##g' /var/jenkins_home/jobs/sandbox/builds/$n/log" | grep -E -i "\(Deploy\)|Approved by|Deploying|Deleted|Pulling from|Digest:|Status:|Production|Finished"
OUTPUT
1.0.23-f344780: digest: sha256:7e74e8d75708279ac534e06c54ac5fec16df26cc655aae8a9002beadc4b92161 size: 856
[Pipeline] { (Deploy)
Deploy this image to production?
Approved by Kwami
Deploying ghcr.io/kwamib/cicd-sandbox-jenkins@sha256:7e74e8d75708279ac534e06c54ac5fec16df26cc655aae8a9002beadc4b92161
Deleted: sha256:7e74e8d75708279ac534e06c54ac5fec16df26cc655aae8a9002beadc4b92161
ghcr.io/kwamib/cicd-sandbox-jenkins@sha256:7e74e8d75708279ac534e06c54ac5fec16df26cc655aae8a9002beadc4b92161: Pulling from kwamib/cicd-sandbox-jenkins
Digest: sha256:7e74e8d75708279ac534e06c54ac5fec16df26cc655aae8a9002beadc4b92161
Status: Downloaded newer image for ghcr.io/kwamib/cicd-sandbox-jenkins@sha256:7e74e8d75708279ac534e06c54ac5fec16df26cc655aae8a9002beadc4b92161
'  Jenkins Production ' is counted as 'jenkins production'
Finished: SUCCESS

The digest the Push stage reported is the digest that was approved, deleted, pulled back and run.

My first version of this stage got one line wrong, and it's worth seeing. It took the first entry in the image's list of digests whole, name and all:

OUTPUT
+ REF=cicd-sandbox@sha256:a9509d6b466d8201f99ab67e980f5d6b47f979b713a39e99afe52ebe65873f87
+ docker pull cicd-sandbox@sha256:a9509d6b466d8201f99ab67e980f5d6b47f979b713a39e99afe52ebe65873f87
Error response from daemon: pull access denied for cicd-sandbox, repository does not exist or may require 'docker login'

The digest was right, but the name in front of it was the local build name, which only ever existed on Jenkins and had just been deleted. A digest says which image; it doesn't say where to find it. That's why the stage now takes only the digest, and puts the registry name in front of it itself.

In my real setup

CivicGrid deploys exactly like this, with the deploy files in their own repository. CI builds and pushes the API image, tagged sha- and the commit. A release is a commit I make in civicgrid-gitops, changing one line in the API's values.yaml, like Deploy civicgrid-api sha-664d196: admin city search accepts city_id (Fix-a-city links). Argo CD watches that repository with automatic sync, prune and selfHeal on, so the cluster follows the file within minutes, and puts things back if anyone changes the cluster by hand. A rollback is git revert of the deploy commit. My hand-written commit is the approval gate: nothing reaches production until I write it.

The one gap this lesson shows up is that CivicGrid deploys by tag, sha-664d196, not by digest. A commit-based tag is unlikely to move, but nothing stops it; a digest can't. Writing the digest into values.yaml would close that.

My StackIT pipeline was the opposite model: Jenkins pushed changes to AWS directly. It used input too, but for the riskiest step, tearing the environment down, not for the deploy. And it ran the app with docker run on the image it had just built locally, and pushed to ECR afterwards. So what ran was a local image that had never been through the registry, which is exactly what this lesson's Jenkins stage avoids.

Spot the problem

The approved latest. A team's deploy job pulls myapp:latest after a manual approval. Reviewers check the run they're approving carefully. What can still go wrong? Between the run and the approval, another merge can move latest, so the reviewer approves one run and production gets another. The deploy has to name the digest that run built.

The rollback that rolled forward. A team keeps its Kubernetes files in the app repository. A bad release goes out, someone reverts the deploy commit, and twenty minutes later production is on a brand-new image nobody asked for. Why? The revert was a push to main, which ran the pipeline, which built and deployed again. Either keep deploy files in their own repository, or have the pipeline ignore changes to them.

The image that only ran on CI. Every gate passes, the deploy succeeds, and the pods in a new ARM-based cluster sit at ImagePullBackOff. What's the first thing to check? Which platforms the image was built for, with docker buildx imagetools inspect. CI built only for its own processor.

Where you are now

Your pipeline now goes all the way. The build job pushes an image for both processors and reports its digest. A production environment holds the deploy until you approve it, and the deploy is a one-line commit that names that digest. The cluster runs exactly the image CI tested and scanned, and a rollback is git revert, with paths-ignore making sure the revert isn't overwritten. In Jenkins, input holds the deploy, and the stage runs the image pulled back from the registry by digest. That's continuous delivery from Lesson 1, finished. Lesson 9 is about the server under all of this: running Jenkins yourself, and keeping it healthy.

Check yourself

  1. Why deploy by digest instead of the latest tag?

  2. The pod said no matching manifest for linux/arm64. What was wrong?

  3. What does environment: production add to the deploy job?

  4. Why didn't the bot's deploy commit start another workflow run?

  5. Why did the rollback need paths-ignore: ['deploy/**']?

  6. Why does the Jenkins condition check env.GIT_BRANCH instead of branch 'main'?

Key terms

  • Continuous delivery: every change is ready to release, and a person decides when.
  • Promote: deploy the same image that passed the earlier steps, rather than building again.
  • Digest: an image's exact ID, sha256:..., which never moves.
  • Platform: the operating system and processor an image is built for, like linux/arm64.
  • QEMU: an emulator that lets one kind of processor build images for another.
  • Environment: a named deploy target in GitHub, with rules like required reviewers.
  • GitOps: running production from files in Git, so a deploy is a commit and a rollback is a revert.
  • Argo CD: a tool in the cluster that keeps it matching a Git repository.
  • CronJob: a Kubernetes object that runs a job on a schedule.
  • paths-ignore: a trigger filter that skips runs when only the listed paths changed.
  • input: the Jenkins step that pauses a build until someone approves it.
  • Rollback: returning production to the version that ran before.

Go deeper

From www.kwamib.dev/resources/cicd/from-image-to-production