Rolling back

By the end you'll know

What a broken release looks like in Kubernetes and in Argo CD, and why the shop can survive one
Why Synced doesn't mean working, and Healthy doesn't mean right
How to roll back with git revert, through a pull request
Why argocd app rollback refuses to run on an automatically synced app
Which way to roll back, and why

Before you start

You need: Lessons 1 to 8 done, with production on release 2 and the root Application managing everything.
Where commands run: your own computer's terminal, in gitops-practice, logged in with argocd login.

The short version

A release that can't start. An image tag that doesn't exist. The new pod failed to pull it, over and over; Argo CD said Synced and Progressing; the two old pods kept serving the whole time.
A release that's wrong. A banner full of typos. It deployed, passed every check, and was Synced and Healthy.
Argo CD's rollback refused: "rollback cannot be initiated when auto-sync is enabled".
git revert fixed both, each through a pull request. A probe on production got an answer to all 224 of its requests, from start to finish.

Releases go wrong

Every release so far has worked. Real ones don't always, and how a team undoes a bad release matters as much as how it ships a good one. There are two kinds of bad, and this lesson tries both:

It can't start: a missing image, a crash at startup, a setting it can't read. Kubernetes can see this.
It starts, and it's wrong: a wrong price, a broken page, a bug. Kubernetes can't see this at all. Only a person, or a test that knows what right looks like, can.

As in Lesson 6, a probe pod asks production for its page twice a second throughout, so "did customers notice?" has a number.

OUTPUT

Synced Healthy a6f6302a172f17f

NAME                        READY   STATUS    RESTARTS   AGE
bookshop-5fbd67c8c7-6dlf6   1/1     Running   0          154m
bookshop-5fbd67c8c7-wxcmf   1/1     Running   0          154m

Production at the start: release 2, two pods.

A release that can't start

Release 3: one value, in production's file, through a pull request.

OUTPUT

--- a/charts/bookshop/values-production.yaml
+++ b/charts/bookshop/values-production.yaml
 image:
-  tag: "1.29"
+  tag: "1.299"

merged: ea4422e Release 3, to production: nginx 1.299 (#30)

1.299 instead of 1.29: a typo, and no such image exists. It skipped staging, too, which is how typos like this usually get through. Watching for a minute:

OUTPUT

after 15s: Synced Progressing ea4422e0524
bookshop-5fbd67c8c7-6dlf6   1/1   Running            0     154m
bookshop-5fbd67c8c7-wxcmf   1/1   Running            0     154m
bookshop-6ff9ccf976-wg74n   0/1   ImagePullBackOff   0     18s

after 30s: Synced Progressing ea4422e0524
...
bookshop-6ff9ccf976-wg74n   0/1   ErrImagePull   0     33s
...
after 60s: Synced Progressing ea4422e0524
bookshop-5fbd67c8c7-6dlf6   1/1   Running        0     155m
bookshop-5fbd67c8c7-wxcmf   1/1   Running        0     155m
bookshop-6ff9ccf976-wg74n   0/1   ErrImagePull   0     63s

Three things to read here:

The new pod can't start. It alternates between ErrImagePull and ImagePullBackOff: Kubernetes tries to download the image, fails, waits a little longer each time, and tries again. The event says why:

OUTPUT

Failed to pull image "nginx:1.299": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.299": ...

The old pods are still running. The rolling update you watched in Lesson 6 starts a new pod first and removes an old one only when the new one is ready. The new one is never ready, so the old ones are never removed. The shop is still on release 2.
Argo CD says Synced, and Progressing. Synced is true: the cluster has exactly what Git says, a Deployment asking for 1.299. It's the Deployment that can't get there. Synced means "matches Git", not "working". Progressing will eventually become Degraded, once the Deployment gives up, which by default takes ten minutes.

And customers?

OUTPUT

 148 200

148 requests, 148 pages. Nobody noticed. The broken release never took a single request, because it never became ready. That's not luck: it's the readiness check from Lesson 3 and the rolling update doing their jobs. Releases that can't start are the safe kind of bad.

Rolling back with git revert

The fix belongs where the mistake is: in Git. git revert makes a new commit that undoes an old one. It doesn't delete anything from history; it adds the opposite change on top.

BASH

git switch -c revert-bad-tag
git revert --no-edit ea4422e
git push -u origin revert-bad-tag
gh pr create --fill --base main
gh pr merge --squash --delete-branch

OUTPUT

merged: 1454906 Revert "Release 3, to production: nginx 1.299 (#30)" (#31)

BASH

git show --stat HEAD

OUTPUT

commit 1454906ce6966e831e592cd0afdf95d04c334c89
...
    Revert "Release 3, to production: nginx 1.299 (#30)" (#31)
    
    This reverts commit ea4422e05247a3727b9e63e89c37783c6d50fe15.

 charts/bookshop/values-production.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

A one-line change, back to 1.29, saying exactly what it undoes. Through a pull request like any other change, so it's reviewed and recorded. Then:

OUTPUT

deployment "bookshop" successfully rolled out
Synced Healthy 1454906ce6966e8

NAME                        READY   STATUS    RESTARTS   AGE
bookshop-5fbd67c8c7-6dlf6   1/1     Running   0          155m
bookshop-5fbd67c8c7-wxcmf   1/1     Running   0          155m

The same two pods, 155 minutes old. The revert put the pod template back to what those pods were made from, ReplicaSet 5fbd67c8c7, so Kubernetes had nothing to replace: it just removed the pod that never started. Healthy again, and the probe still answered every request.

A release that works, and is wrong

The second kind. A new banner, merged:

OUTPUT

-  banner: Free delivery on every order this week.
+  banner: Free delivery on evry order this weak.

merged: c8e2a6e Bookshop: a new banner (#32)
deployment "bookshop" successfully rolled out
Synced Healthy c8e2a6e1c5a61ca

<div class="banner">Free delivery on evry order this weak.</div>

Synced, Healthy, and wrong. Every check passed: the pods started, the readiness check got a page, Argo CD matched Git. None of them can tell that "evry" and "weak" are wrong. Healthy means "running and answering", not "right". This is the kind of bad release that reaches customers, and the kind a rollback is really for.

Argo CD's own rollback

Argo CD keeps a history of deploys, and has a command to go back to one. The deploy before the typo was 21:

BASH

argocd app history bookshop | tail -4
argocd app rollback bookshop 21

OUTPUT

19      2026-11-04 12:57:09 -0500 EST  main (823683e)
20      2026-11-04 15:31:32 -0500 EST  main (ea4422e)
21      2026-11-04 15:32:45 -0500 EST  main (1454906)
22      2026-11-04 15:32:58 -0500 EST  main (c8e2a6e)

{"level":"fatal","msg":"rpc error: code = FailedPrecondition desc = rollback cannot be initiated when auto-sync is enabled","time":"2026-11-04T15:33:01-05:00"}

Refused. "Rollback cannot be initiated when auto-sync is enabled." And it's right to refuse. An Argo CD rollback puts the cluster back to an older commit, but Git still says the typo. With automated sync, Argo CD would see the cluster no longer matching Git and sync it straight back, typo and all. To use it, you'd have to switch automated sync off first; and since Lesson 8, the root would switch it back on within a second. With GitOps done fully, the only rollback that sticks is the one in Git.

argocd app rollback still has a place: on an app with manual sync, as an emergency brake while the fix is being made in Git. But Git has to catch up afterwards, or the next sync brings the bad version back.

Rolling back, again

So, git revert, the same way:

OUTPUT

merged: 75fc471 Revert "Bookshop: a new banner (#32)" (#33)
deployment "bookshop" successfully rolled out
Synced Healthy 75fc4716bb1ab2d

<div class="banner">Free delivery on every order this week.</div>

Fixed, through the same review as any other change, and Argo CD's history shows it deployed 12 seconds after the typo.

What the history says

BASH

git log --oneline -5

OUTPUT

75fc471 Revert "Bookshop: a new banner (#32)" (#33)
c8e2a6e Bookshop: a new banner (#32)
1454906 Revert "Release 3, to production: nginx 1.299 (#30)" (#31)
ea4422e Release 3, to production: nginx 1.299 (#30)
a6f6302 Bookshop: remove the preview copy, and what it deployed (#29)

Nothing was hidden. Both bad releases are there, each followed by the revert that undid it. A month from now, "what happened on the 4th?" has a complete answer, and so does "has this mistake happened before?". Argo CD's history tells the same story, deploy by deploy:

OUTPUT

20      2026-11-04 15:31:32 -0500 EST  main (ea4422e)
21      2026-11-04 15:32:45 -0500 EST  main (1454906)
22      2026-11-04 15:32:58 -0500 EST  main (c8e2a6e)
23      2026-11-04 15:33:10 -0500 EST  main (75fc471)

And the probe, over the whole lesson:

OUTPUT

 224 200

224 requests, 224 pages, through a broken release, two rollouts and two reverts. Lesson 6 measured the odd dropped request during rollouts and added the preStop pause; this time, with two full rollouts, none were dropped. One run isn't proof, but it's the measurement you'd want to keep seeing.

In real systems

Revert first, investigate second. The fastest safe fix for a bad release is usually reverting it, not fixing forward. Once production is back to known-good, there's time to understand what went wrong.
Staging catches the second kind. The typo would have been obvious on staging's page. Many teams make promotion to production require a person to look, or automated tests that know what right looks like: a check that the page has a price, that the API returns books.
Progressive delivery. Tools like Argo Rollouts send a small share of traffic to a new release first, watch its error rate, and roll back automatically if it rises. That catches some "works but wrong" releases, the ones that show up as errors.
Degraded should page someone. Argo CD's notifications, or an alert on its metrics in the Monitoring track, turn "Progressing for ten minutes" into a message, so a stuck release doesn't sit unnoticed.

Spot the problem

Work out what's wrong before you read the explanation.

The rollback that came back. A bad release reaches production at night. The engineer on call can't wait for a review, so they switch the app to manual sync in the web interface and run argocd app rollback to the previous deploy. It works: the shop is fixed. They go back to bed, planning to revert in Git in the morning. In the morning, someone else switches automated sync back on, because "it's always on". Within seconds the bad release is back. What went wrong? Git still said the bad release. The rollback changed the cluster, not the description of it, so the moment automated sync returned, Argo CD synced to Git, as it should. (And with an app-of-apps root, the switch to manual sync would have been undone in a second anyway.) The rollback was a valid emergency brake; the mistake was not following it straight away with a git revert, so that Git and the cluster agree again. In GitOps, a fix isn't finished until it's in Git.

Keep this

To	Run
See why a pod won't start	`kubectl get pods`, then `kubectl get events --field-selector reason=Failed`
Undo a commit, keeping history	`git revert --no-edit COMMIT`
Roll back, the GitOps way	A branch, `git revert`, a pull request, merged
See what a revert changed	`git show --stat HEAD`
Argo CD's deploy history	`argocd app history bookshop`
Argo CD's rollback, manual sync only	`argocd app rollback bookshop ID`, then revert in Git
Check what production is serving	`kubectl exec -n bookshop deploy/bookshop -- grep ...`
Count what customers got	A probe pod, then `kubectl logs probe \| awk '{print $2}' \| sort \| uniq -c`

Where you are now

You've broken production twice, and put it back both times with a pull request, without a customer noticing. You know Synced isn't working and Healthy isn't right, and why the rollback that sticks is the one in Git. One kind of setting has stayed out of Git so far: secrets. Lesson 10 puts a database password into the GitOps repository without anyone being able to read it.

Check yourself

The new pod couldn't pull nginx:1.299. Why did the shop keep working?
During the broken release, Argo CD said Synced and Progressing. What did Synced mean?
The banner with typos was Synced and Healthy. Why didn't any check catch it?
Why did argocd app rollback refuse to run?
What does git revert do?
After the revert, why didn't the production pods change?

Key terms

Rollback: returning to an earlier, known-good release.
git revert: a new commit that undoes an earlier one, keeping both in history.
ImagePullBackOff: Kubernetes waiting longer and longer between failed attempts to download an image.
ErrImagePull: a failed attempt to download an image.
Progressing: Argo CD's health for an app that's still on its way to the state Git asks for.
Degraded: Argo CD's health for an app that's failed to get there.
Fix forward: fixing a bad release with a new change, rather than reverting it.
Progressive delivery: releasing to a small share of traffic first, and rolling back automatically if it misbehaves.

Go deeper

The Argo CD documentation: argocd app rollback (opens in a new tab), and Argo Rollouts (opens in a new tab).
Try: make a release that crashes instead, by changing the image to one that exits at once, like busybox:1.37 with no command. What do the pods and Argo CD's health say, and does the shop stay up?
Try: lower the Deployment's progressDeadlineSeconds to 60 in the chart, push a bad tag to staging, and watch how long until Argo CD says Degraded.