How to use this page
Find the job you're trying to do, and copy the pattern. Words in capitals, like APP, NAMESPACE or FILE, are yours to replace. Every row links to the lesson that explains it, with real output from the practice cluster. Use the print button at the top to keep a copy next to you.
Three habits that make everything here easier:
- Change Git, not the cluster. With self-heal on, a change made with
kubectlorargocd app setis undone within seconds. A branch, a commit and a pull request is how anything changes, including a rollback. - Refresh instead of waiting. Argo CD checks Git on a timer, every few minutes.
argocd app get APP --refreshchecks now. - Synced isn't working, and Healthy isn't right. Synced means the cluster matches Git. Healthy means the pods are running and answering. Neither can tell you the page is correct: look at it.
When Argo CD says no
Start here when something's wrong. Match the message or the symptom, then look where it says.
| The message or symptom | It means | Look at | Lesson |
|---|---|---|---|
Failed to establish connection to localhost:8080 ... connection refused, from argocd | The port-forward isn't running, so the argocd command can't reach the server | Start kubectl port-forward svc/argocd-server -n argocd 8080:443 in its own terminal, then log in again | 2, 10 |
The connection to the server localhost:8080 was refused, from kubectl | kubectl has no cluster to talk to: the cluster is deleted or stopped. It falls back to that address | kind get clusters, docker start gitops-control-plane gitops-worker | 1, 11 |
SSL certificate problem: self-signed certificate | Argo CD made its own certificate. Encrypted, but nothing has vouched for it | --insecure on a local port-forward only | 2 |
OutOfSync, and it stays that way | Self-heal or automated sync is off, so drift is reported and left | argocd app diff APP, then argocd app sync APP | 2, 4 |
OutOfSync, with one resource Git doesn't have | Prune is off, so Argo CD won't delete it | argocd app sync APP --prune, or prune: true | 4 |
Merged, but still Synced to the old commit | Argo CD hasn't checked Git yet. Or the commit never reached GitHub | argocd app get APP --refresh, and git status for "ahead of origin" | 3 |
Synced, but the pods serve the old page | A ConfigMap mounted as files updates in place, about a minute later, without new pods | A checksum/ annotation on the pod template, so a new page rolls the pods | 4, 6 |
Synced and Progressing, with ImagePullBackOff or ErrImagePull | The new pod can't download its image: often a tag that doesn't exist. The old pods keep serving | kubectl get events -n NAMESPACE --field-selector reason=Failed, then git revert | 9 |
Synced and Healthy, and wrong | Every check passed. They test running, not right | The page itself. Revert in Git | 9 |
rollback cannot be initiated when auto-sync is enabled | argocd app rollback would be undone by the next sync from Git | git revert COMMIT, by pull request | 9 |
An argocd app set change disappears in a second | The root Application manages this Application's file, and put it back | Edit the file in argocd/, by pull request | 8 |
| Application deleted, its pods still running | No finalizer, so deleting the Application left its resources behind | kubectl delete namespace NAMESPACE, and resources-finalizer.argocd.argoproj.io next time | 8 |
helm template accepts a wrong value, or ignores a misspelled one | Helm doesn't check values unless the chart has a schema | values.schema.json in the chart | 5 |
is not a valid chart repository or cannot be reached ... 404 Not Found | The chart repository's address is wrong, or the project moved it | The project's current install instructions | 10 |
targetRevision: null in an Application | Something wrote an empty value into the file | The script that wrote it: stop on empty values | 10 |
ErrUnsealFailed ... no key could decrypt secret | The SealedSecret was sealed for another namespace or name, or for another cluster's key | kubectl get events -n NAMESPACE --field-selector involvedObject.kind=SealedSecret | 10, 11 |
OutOfSync and Missing right after a rebuild | It needs something another app hasn't installed yet, such as a SealedSecret's definition. It catches up on its own | Wait, and watch kubectl get applications -n argocd | 11 |
Sync and health
Every Application has two statuses. They answer different questions.
| Status | Means | Lesson |
|---|---|---|
Synced | The cluster matches what Git says. Not the same as working | 1, 9 |
OutOfSync | The cluster differs from Git: a new commit, or a change by hand | 1, 2 |
Healthy | What's running is ready and answering. Not the same as right | 1, 9 |
Progressing | On its way: pods starting, or stuck starting. Becomes Degraded if it gives up | 1, 9 |
Degraded | It failed to get there: by default after ten minutes of a stuck rollout | 9 |
Missing | Git has it and the cluster doesn't, yet | 11 |
The cluster and Argo CD
| To | Run | Lesson |
|---|---|---|
| Create the practice cluster | kind create cluster --name gitops --config kind-gitops.yaml | 1 |
| Install a pinned Argo CD | kubectl apply -n argocd --server-side --force-conflicts -f .../v3.5.7/manifests/install.yaml | 1 |
| Wait for it to be ready | kubectl wait -n argocd --for=condition=Available deployment --all | 1 |
| Pause the cluster, start it again | docker stop gitops-control-plane gitops-worker, then docker start | 1 |
| Delete the cluster | kind delete cluster --name gitops | 1, 11 |
The argocd command
| To | Run | Lesson |
|---|---|---|
| Install it | brew install argocd | 2 |
| Reach the server, in its own terminal | kubectl port-forward svc/argocd-server -n argocd 8080:443 | 2 |
| Read the first password | kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d | 2 |
| Log in | argocd login localhost:8080 --username admin --password "$(cat .argocd-admin)" --insecure | 2 |
| Change a password | argocd account update-password --account admin | 2, 11 |
| Both versions | argocd version --short | 2 |
| Every app at once | argocd app list | 2, 7 |
| An app's details and resources | argocd app get APP | 2 |
| Check Git now | argocd app get APP --refresh | 3, 4 |
| How the cluster differs from Git | argocd app diff APP; exit code 1 if different | 2 |
| The YAML it applies | argocd app manifests APP | 2 |
| What it deployed, and when | argocd app history APP | 2, 9 |
| Sync by hand | argocd app sync APP | 2, 4 |
| The web interface | https://localhost:8080, with the port-forward running | 2 |
Applications
| To | Run | Lesson |
|---|---|---|
| Point one at your repository | repoURL:, targetRevision: main, path: FOLDER | 3 |
| Add one, before app-of-apps | kubectl apply -f argocd/APP.yaml | 1, 3 |
| See its status with kubectl | kubectl get application APP -n argocd | 1 |
| Which commit is running | kubectl get application APP -n argocd -o jsonpath='{.status.sync.revision}' | 1, 3 |
| What Argo CD did | kubectl get events -n argocd --field-selector involvedObject.name=APP | 1 |
| How often it checks Git | timeout.reconciliation in the argocd-cm ConfigMap; blank is the default | 3 |
| Deploy a change | A branch, a commit, a pull request, merged | 3 |
| Reach an app | kubectl port-forward svc/APP -n NAMESPACE 8081:80 | 3 |
Sync policies
| To | Run | Lesson |
|---|---|---|
| Apply Git changes without a person | syncPolicy: automated: {} | 4 |
| Delete what Git no longer has | prune: true | 4 |
| Undo hand changes to the cluster | selfHeal: true | 4 |
| Sync by hand, and prune | argocd app sync APP --prune | 4 |
| Never prune one resource | argocd.argoproj.io/sync-options: Prune=false, as an annotation | 4 |
| Let something else own a field | ignoreDifferences: in the Application | 4 |
| Create the namespace if needed | syncOptions: [CreateNamespace=true] | 3 |
| Quick changes, for testing only | argocd app set APP --self-heal=false, --sync-policy none | 2, 4 |
Helm
| To | Run | Lesson |
|---|---|---|
| Check a chart is well formed | helm lint charts/bookshop | 5 |
| See the YAML it makes | helm template NAME charts/bookshop | 5 |
| One template only | helm template NAME charts/bookshop -s templates/FILE.yaml | 10 |
| Change one value | --set replicas=3, or --set-string for text | 5 |
| Use values files, later winning | -f values.yaml -f values-production.yaml | 5, 7 |
| Compare two renders | diff <(helm template ...) <(helm template ... -f FILE) | 5, 7 |
| Read a value | {{ .Values.replicas }} | 5 |
| Paste a block, indented | {{- toYaml .Values.resources | nindent 12 }} | 5 |
| Reuse a piece | define in _helpers.tpl, then include "NAME" . | nindent 4 | 5 |
| Read a file, as a template | {{ tpl (.Files.Get "files/index.html") . }} | 5 |
| Only when a value is set | {{- if .Values.shop.banner }} ... {{- end }} | 7, 10 |
| Install, upgrade, list, remove | helm install, helm upgrade, helm list -n NAMESPACE, helm uninstall | 5 |
| History, and roll back, without Argo CD | helm history NAME, helm rollback NAME REVISION | 5 |
Helm in Argo CD
| To | Run | Lesson |
|---|---|---|
| Point an Application at a chart in Git | path: charts/bookshop, with a Chart.yaml in it | 6 |
| Name the release, choose values | helm: releaseName: bookshop, valueFiles: [values.yaml, values-production.yaml] | 6, 7 |
| A chart from a chart repository | repoURL: URL, chart: NAME, targetRevision: CHART-VERSION | 10 |
| Set a chart's values in the Application | helm: valuesObject: | 10 |
| Find a chart's versions | helm repo add NAME URL, then helm search repo NAME/CHART | 10 |
| Preview a change against what's running | helm template ... > FILE, then kubectl diff -n NAMESPACE -f FILE | 6 |
| Check there's no Helm release | helm list -A: Argo CD renders charts, and doesn't install them as releases | 6 |
| Roll pods when a ConfigMap changes | checksum/page: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} | 6 |
| Pause before a pod stops | lifecycle: preStop: exec: command: ["sleep", "5"] | 6 |
| Watch a rollout finish | kubectl rollout status deployment/APP -n NAMESPACE | 6, 7 |
| Probe a Service from inside the cluster | kubectl run probe --image=curlimages/curl:8.11.1 ... | 6, 9 |
| Count the probe's answers | kubectl logs probe | awk '{print $2}' | sort | uniq -c | 6, 9 |
Environments and promotion
| To | Run | Lesson |
|---|---|---|
| Give an environment its own settings | values-staging.yaml, values-production.yaml in the chart | 7 |
| A second environment | A second Application: a new name, values file and namespace | 7 |
| Check an image tag exists | docker buildx imagetools inspect nginx:TAG | 7 |
| See what a chart change does to production | Render production before and after, and diff | 7 |
| Promote a release | Copy the release's values into production's file, by pull request | 7 |
| Check production asks for what staging ran | diff the two files, ignoring the environment's own settings | 7 |
| Check after a rollout has settled | kubectl rollout status, then kubectl exec deploy/APP -- ... | 7 |
App-of-apps
| To | Run | Lesson |
|---|---|---|
| A root Application | An Application with path: argocd and destination.namespace: argocd | 8 |
| Start the chain, once | kubectl apply -f bootstrap/root.yaml | 8, 11 |
| Check files match before adopting | kubectl diff -f argocd/FILE.yaml | 8 |
| See what the root manages | argocd app get root | 8 |
| Who owns an Application | Its argocd.argoproj.io/tracking-id annotation | 8 |
| Add, or remove, an app | Add, or delete, a file in argocd/, by pull request | 8 |
| Delete an app's resources with it | finalizers: [resources-finalizer.argocd.argoproj.io] | 8 |
| Clean up orphans by hand | kubectl delete namespace NAMESPACE | 8 |
Rolling back
| To | Run | Lesson |
|---|---|---|
| See why a pod won't start | kubectl get pods, then kubectl get events --field-selector reason=Failed | 9 |
| Undo a commit, keeping history | git revert --no-edit COMMIT | 9 |
| Roll back, the GitOps way | A branch, git revert, a pull request, merged | 9 |
| See what a revert changed | git show --stat HEAD | 9 |
| Argo CD's rollback, manual sync only | argocd app rollback APP ID, then revert in Git | 9 |
Secrets
| To | Run | Lesson |
|---|---|---|
| Read a Secret's value | kubectl get secret NAME -n NAMESPACE -o jsonpath='{.data.KEY}' | base64 -d | 10 |
| Install kubeseal | brew install kubeseal | 10 |
| Get the public key | kubeseal --fetch-cert | 10 |
| Seal one value | printf %s "VALUE" | kubeseal --raw --namespace NAMESPACE --name NAME | 10 |
| Check it unsealed | kubectl get sealedsecret,secret -n NAMESPACE | 10 |
| See why it didn't | kubectl get events -n NAMESPACE --field-selector involvedObject.kind=SealedSecret | 10 |
| Check a password isn't in Git | git log -p | grep -c 'PASSWORD' | 10 |
| Back up the sealing key | kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml, to a file in .gitignore | 10 |
| Keep a file out of Git | Add it to .gitignore before it exists, and chmod 600 it | 2, 10 |
Rebuilding from Git
The order matters. Everything after the root comes from Git.
| Step | Run | Lesson |
|---|---|---|
| 1. A cluster | kind create cluster --name gitops --config kind-gitops.yaml | 11 |
| 2. The sealing key, before Sealed Secrets | sed -E '/^ +(resourceVersion|uid|creationTimestamp):/d' the backup, then kubectl apply -f it | 11 |
| 3. Argo CD | kubectl create namespace argocd, then the pinned install | 1, 11 |
| 4. The admin password | argocd account update-password --account admin, then delete argocd-initial-admin-secret | 2, 11 |
| 5. The root | kubectl apply -f bootstrap/root.yaml | 8, 11 |
| 6. Watch it come back | kubectl get applications -n argocd | 11 |
| If the key came too late | kubectl delete pod -n kube-system -l app.kubernetes.io/name=sealed-secrets | 11 |
Git rebuilds what runs. Data an app stores, in a database or on a disk, needs backups of its own. See Lesson 11.