Rebuilding the cluster from Git

The claim, tested

Lesson 1 made a claim: with GitOps, Git describes the cluster, and the cluster is just Git made real. If that's true, losing the cluster shouldn't matter. Every lesson since has moved one more thing into Git: the Applications, the chart, each environment's values, the root, a sealed password. This lesson tests the claim the only honest way: delete the cluster, and see what comes back.

First, a record of what's there now:

$ apps
bookshop           Synced   Healthy
bookshop-staging   Synced   Healthy
guestbook          Synced   Healthy
root               Synced   Healthy
sealed-secrets     Synced   Healthy

$ banner
<div class="banner">Free delivery on every order this week.</div>

$ kubectl exec -n bookshop deploy/bookshop -- printenv DB_PASSWORD
f994119e3a31e48126d04262

$ kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
NAME                      TYPE                DATA   AGE
sealed-secrets-keygzzt8   kubernetes.io/tls   2      36m

$ git log --oneline -1
dd5a1f4 Bookshop: the sealed password, for production (#38)

apps is a short kubectl get applications that prints each app's sync and health, and banner greps the page for its banner. Five apps, the release 2 banner, the password from Lesson 10, and the key that unseals it. Production is at commit dd5a1f4.

And the only things brought along, besides Git:

-rw-------  1 mayowababatola  staff    25 Nov  3 16:25 .argocd-admin
-rw-------  1 mayowababatola  staff  7085 Nov  4 17:38 .sealed-secrets-key.yaml

Deleting the cluster

kind delete cluster --name gitops
kubectl get nodes

Deleting cluster "gitops" ...
Deleted nodes: ["gitops-control-plane" "gitops-worker"]

The connection to the server localhost:8080 was refused - did you specify the right host or port?

Gone: both nodes, every pod, Argo CD and its history, the Secret holding the password. The error is kubectl with no cluster left to talk to. It falls back to a default address, localhost:8080, which happens to be the port Argo CD's port-forward used, but has nothing to do with it.

By hand: a cluster, a key, Argo CD

A few things can't come from Argo CD, because Argo CD isn't there yet. First, a new cluster, from the file in Git:

kind create cluster --name gitops --config kind-gitops.yaml
kubectl get namespaces

NAME                 STATUS   AGE
default              Active   10s
kube-node-lease      Active   10s
kube-public          Active   10s
kube-system          Active   10s
local-path-storage   Active   6s

elapsed: 20s

Empty, 20 seconds after the delete. Then the sealing key, before anything else:

sed -E '/^ +(resourceVersion|uid|creationTimestamp):/d' .sealed-secrets-key.yaml > /tmp/sealed-key-restore.yaml
kubectl apply -f /tmp/sealed-key-restore.yaml

secret/sealed-secrets-keygzzt8 created

The sed removes three fields the old cluster stamped on the Secret when it was made: its version, its ID and its creation time. They belonged to the old cluster, and Kubernetes won't create something that claims them. The key goes back first because of what Sealed Secrets does when it starts: it looks for existing keys, and only makes a new one if there are none. Put the old key there first, and the controller uses it. Install the controller first, and it makes a new key that can't read anything in Git.

Then Argo CD, exactly as in Lesson 1, at the same pinned version:

kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.7/manifests/install.yaml
kubectl rollout status deployment/argocd-server -n argocd --timeout=300s

And the admin password back to the one in .argocd-admin, as in Lesson 2, with the first one deleted:

$ argocd account update-password --account admin --current-password "$INITIAL" --new-password "$(cat .argocd-admin)"
Password updated

$ argocd login localhost:8080 --username admin --password "$(cat .argocd-admin)" --insecure
'admin:login' logged in successfully

$ kubectl delete secret argocd-initial-admin-secret -n argocd
secret "argocd-initial-admin-secret" deleted

$ apps
(0 Applications)

elapsed: 91s

Argo CD, running, knowing about nothing. 91 seconds since the delete.

By Git: one apply

The last thing by hand, and the one Lesson 8 said would be:

kubectl apply -f bootstrap/root.yaml

Then nothing but watching, every 10 seconds:

after 10s:
bookshop           OutOfSync   Missing
bookshop-staging   Synced      Progressing
guestbook          Synced      Progressing
root               Synced      Healthy
sealed-secrets     Synced      Progressing

after 20s:
bookshop           Synced   Progressing
bookshop-staging   Synced   Healthy
guestbook          Synced   Progressing
root               Synced   Healthy
sealed-secrets     Synced   Progressing

...

after 50s:
bookshop           Synced   Progressing
bookshop-staging   Synced   Healthy
guestbook          Synced   Progressing
root               Synced   Healthy
sealed-secrets     Synced   Healthy

from the root apply: 51s; from the delete: 142s

In 10 seconds, the root had read argocd/ and made all four other Applications, and they were already deploying. At 50 seconds the bookshop was the last one still Progressing, and the check a moment later counted all five Synced and Healthy. 142 seconds from deleting the cluster to having all of it back, and most of those were spent installing Argo CD.

Look at bookshop at 10 seconds, though: OutOfSync, Missing, the only app not yet Synced. Production's chart includes a SealedSecret, and a SealedSecret is a kind of resource Kubernetes only knows about once the Sealed Secrets chart has installed its definition. The two apps were syncing at the same moment, so most likely the bookshop's first attempt came before that definition existed. Ten seconds later it was Synced. Then it waited on something else:

NAME                                   STATUS   SYNCED   AGE
sealedsecret.bitnami.com/bookshop-db            True     40s

NAME                 TYPE     DATA   AGE
secret/bookshop-db   Opaque   1      3s

The SealedSecret arrived 40 seconds before the check; the Secret made from it, 3 seconds before. In between, the controller was still starting. The pods were created at the start too, but they need that Secret for DB_PASSWORD, so they couldn't start until it existed, which is why bookshop stayed Progressing the longest. Nobody had to order anything. Each piece kept trying until what it needed was there, and the whole system settled. That's the controller pattern from Lesson 1, across five apps at once.

Is it the same shop?

$ kubectl get pods -n bookshop
NAME                        READY   STATUS    RESTARTS   AGE
bookshop-74d9f67bc8-4kf4c   1/1     Running   0          40s
bookshop-74d9f67bc8-55btt   1/1     Running   0          40s

$ banner
<div class="banner">Free delivery on every order this week.</div>

$ kubectl exec -n bookshop deploy/bookshop -- printenv DB_PASSWORD
f994119e3a31e48126d04262

same password as before the delete: yes

$ kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
NAME                      TYPE                DATA   AGE
sealed-secrets-keygzzt8   kubernetes.io/tls   2      2m2s

The same shop. Two pods, new and 40 seconds old, but from ReplicaSet 74d9f67bc8, the same as in Lesson 10: the same pod template gives the same name, in any cluster. The same banner. The same password, unsealed from the same 740 characters in Git, by the same key, sealed-secrets-keygzzt8, now 2 minutes old. And only that key: the controller found it, and didn't make another. Staging and the guestbook were back too, one pod each.

Some things didn't come back, and shouldn't have:

$ argocd app history bookshop
ID      DATE                           REVISION
0       2026-11-04 18:16:06 -0500 EST  main (dd5a1f4)

$ git status --short; git log --oneline -1
dd5a1f4 Bookshop: the sealed password, for production (#38)

Argo CD's history starts again at 0: it was a record kept in the old cluster, not in Git. Git's history is where every release and every revert from Lessons 3 to 10 is still recorded. And Git didn't change at all: nothing was committed to rebuild. The cluster went back to matching what Git already said.

What Git can't rebuild

The bookshop was easy to rebuild for one reason: it stores nothing. Its page is in Git, its settings are in Git, and its password is in Git, sealed. A real shop has orders, customers, accounts: data, kept in a database, on a disk. Delete the cluster and that's gone, and no repository brings it back. GitOps rebuilds what runs. What's stored needs backups: copies kept somewhere else, and restored once in a while to prove they work. Git would rebuild the database server; only a backup brings back what was in it.

And the list of what came from outside Git, in order:

Five steps, two secret files, and everything else from Git.

Spot the problem

Work out what's wrong before you read the explanation.

The key that came too late. Someone rebuilds the cluster in a hurry: a new cluster, Argo CD, the root. Everything comes up, except the bookshop, stuck Progressing, with its pods waiting and the SealedSecret's events saying ErrUnsealFailed ... no key could decrypt secret. Then they remember the key, apply the backup, and wait. Nothing changes. What went wrong? The controller started before the key was there, found no keys, and made a new one. A new key can't decrypt values sealed with the old one. Applying the backup afterwards puts the old key in the cluster, but the controller loads its keys when it starts, so it needs restarting to read it: kubectl delete pod -n kube-system -l app.kubernetes.io/name=sealed-secrets makes Kubernetes start a new controller pod, which finds both keys. Better, as in this lesson: restore the key before anything that uses it exists. Then take the backup again: the cluster now has two keys, and anything sealed in the meantime used the new one.

Where you are now

That's the track. You've watched Argo CD keep a cluster matching Git, undo changes made by hand, deploy a Helm chart three ways from one set of templates, promote a release by changing one value, manage its own Applications from a folder, roll back with a pull request, keep a password in Git without anyone being able to read it, and rebuild everything from scratch in under two and a half minutes. Git is the source of truth, and you've tested it. The Monitoring track runs on this same cluster, installed the same way: Prometheus, Grafana and Loki, from Helm charts, through Argo CD.

Check yourself

Key terms

• Source of truth: the one place that says what should exist. Here, Git.
• Bootstrap: the first steps, done by hand, that let the automation take over.
• Disaster recovery: getting a system back after losing it.
• Disaster recovery drill: doing that on purpose, to find what's missing before it matters.
• Stateless: an app that stores nothing itself, like the bookshop, and so is easy to rebuild.
• Stateful: an app that stores data, which needs backups as well as Git.

Go deeper

• The Argo CD documentation: Cluster bootstrapping (opens in a new tab) and Sync waves (opens in a new tab), a way to make one app wait for another on purpose.
• Try: rebuild again, but install everything before restoring the key, as in Spot the problem. What do the SealedSecret's events say, and does deleting the controller's pod fix it?
• Try: write the whole rebuild as one script, bootstrap/rebuild.sh, and time it. What's the slowest step, and could any of it move into Git?