Helm charts in Argo CD

From folder to chart

Lesson 5 made the chart, and Lesson 4's bookshop still runs from apps/bookshop/. Switching over is a change to one file, the Application. But it's a change to a running shop, so first: what would actually change?

helm template bookshop charts/bookshop > /tmp/bookshop-chart.yaml
kubectl diff -n bookshop -f /tmp/bookshop-chart.yaml

kubectl diff compares files with what's running, and changes nothing. The chart, rendered, against the live bookshop (trimmed to the changed lines):

diff ... apps.v1.Deployment.bookshop.bookshop
-    argocd.argoproj.io/tracking-id: bookshop:apps/Deployment:bookshop/bookshop
+    app.kubernetes.io/instance: bookshop
+    app.kubernetes.io/name: bookshop
+    app.kubernetes.io/version: "1"
+    helm.sh/chart: bookshop-0.1.0
+        app.kubernetes.io/instance: bookshop
+        app.kubernetes.io/name: bookshop
+        app.kubernetes.io/version: "1"
+        helm.sh/chart: bookshop-0.1.0
diff ... v1.ConfigMap.bookshop.bookshop-page
-    argocd.argoproj.io/tracking-id: bookshop:/ConfigMap:bookshop/bookshop-page
+  labels:
+    app: bookshop
...
diff ... v1.Service.bookshop.bookshop
-    argocd.argoproj.io/tracking-id: bookshop:/Service:bookshop/bookshop
+  labels:
...

Only labels. The + lines are the standard labels the chart adds. The - lines are Argo CD's tracking annotation from Lesson 2, which helm template knows nothing about; Argo CD adds it back when it applies. Same names, same page, same selector: a safe switch. The second set of + lines is inside the pod template, which matters in a moment.

The Application, pointed at the chart

  source:
    repoURL: https://github.com/Kwamib/gitops-practice.git
    targetRevision: main
    path: charts/bookshop
    helm:
      releaseName: bookshop
      valueFiles:
        - values.yaml

• path: charts/bookshop: Argo CD sees a Chart.yaml there, and treats the folder as a Helm chart.
• releaseName: bookshop: what .Release.Name is in the templates. Without it, Argo CD would use the Application's name, which happens to be the same here; setting it says so.
• valueFiles: which values files to use, in order, from the chart's folder. Lesson 7 adds a second one.

Through a pull request, then applied, as the Application isn't watched yet:

kubectl apply -f argocd/bookshop.yaml
kubectl rollout status deployment/bookshop -n bookshop
kubectl get pods -n bookshop -l app=bookshop

merged: 445401e Bookshop: deploy from the Helm chart (#10)
application.argoproj.io/bookshop configured
synced to 445401e, Healthy, after 4s
deployment "bookshop" successfully rolled out

NAME                        READY   STATUS    RESTARTS   AGE
bookshop-6b658ddf64-2bmlt   1/1     Running   0          3s
bookshop-6b658ddf64-s477c   1/1     Running   0          4s

New pods, from a new ReplicaSet, 6b658ddf64. Those labels inside the pod template changed the template, so Kubernetes rolled the Deployment: start a new pod, wait for its readiness check, stop an old one, repeat. And Argo CD's view of the app:

Source:
- Repo:             https://github.com/Kwamib/gitops-practice.git
  Target:           main
  Path:             charts/bookshop
  Helm Values:      values.yaml
Sync Status:        Synced to main (445401e)
Health Status:      Healthy

No release

Lesson 5's helm install left a release, with a Secret per revision. Argo CD's chart?

helm list -A
kubectl get secrets -n bookshop

NAME	NAMESPACE	REVISION	UPDATED	STATUS	CHART	APP VERSION

No resources found in bookshop namespace.

Nothing, in any namespace. Argo CD renders the chart, like helm template, and applies the result like any other YAML. Helm never "installs" anything. So helm upgrade and helm rollback don't apply to apps Argo CD manages, and that's fine: the history is argocd app history and git log, as it has been since Lesson 3. One rule follows from this: never run helm install or helm upgrade on an app Argo CD manages. It would fight Argo CD over the same resources, just like a kubectl edit.

Fixing the ConfigMap delay

Lessons 3 and 4 found that a changed page took 70 to 80 seconds to reach the pods, because changing a ConfigMap doesn't restart anything. The standard Helm fix is one annotation in the pod template:

  template:
    metadata:
      annotations:
        # Changes whenever the page changes, so a new page rolls the pods.
        checksum/page: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

It renders the ConfigMap template, and puts a SHA-256 checksum of it in an annotation: a short fingerprint that changes if a single character of the page does. Change the page, and the checksum changes; the checksum is in the pod template, so the pod template changes; and a changed pod template means a rollout, with new pods that read the new page as they start. Adding it rolled the pods once, as expected. Then the test, a new headline, changed only in values.yaml:

-  headline: Books for people who run things.
+  headline: Books for people who keep things running.

merged: cd2ef75 Bookshop: a new headline (#12)
synced to cd2ef75, Healthy, after 4s
deployment "bookshop" successfully rolled out
headline in every pod 13s after the merge started:

bookshop-5ffd47c5d6-8h7hg: <h1>Books for people who keep things running.</h1>
bookshop-5ffd47c5d6-cpx8h: <h1>Books for people who keep things running.</h1>

13 seconds from merge to every pod, counting the pull request itself. The stale minute is gone, because the pods with the old page no longer exist. And the change was one line in a values file: the chart, the page and the Application didn't change at all.

Zero downtime?

Three rollouts so far, and the shop looked fine throughout. But "looked fine" isn't a measurement. While those rollouts ran, a probe pod in the same namespace asked the shop for its page twice a second, and kept the answer:

kubectl run probe -n bookshop --image=curlimages/curl:8.11.1 --restart=Never --command -- \
  sh -c 'while true; do echo "$(date +%T) $(curl -s -o /dev/null -m 2 -w "%{http_code}" http://bookshop/)"; sleep 0.5; done'

curl prints the HTTP status: 200 for the page, or 000 if there was no answer at all within 2 seconds. Its verdict on those three rollouts:

kubectl logs probe -n bookshop | awk '{print $2}' | sort | uniq -c

   3 000
  74 200

Three requests out of 77 got no answer. Small, and only during rollouts, but for a real shop that's three customers with an error page. Kubernetes' rolling update waits for new pods to be ready before stopping old ones, so where do failures come from?

The usual answer is a race at the moment a pod stops. When Kubernetes decides an old pod should go, two things start at once: the pod is told to shut down, and the Service is told to stop sending it requests. Those happen in different parts of the cluster, and the second can take a moment longer. In that moment, a request can still be sent to a pod that's already shutting down. The standard fix is to make the pod wait before it starts shutting down: a preStop hook.

          lifecycle:
            preStop:
              # Keep answering for a few seconds after being told to stop, while the
              # Service catches up and stops sending this pod new requests.
              exec:
                command: ["sleep", "5"]

Does it work? Measured properly: the probe four times a second, and two identical headline changes, one with the hook and one without, with every failed request's time:

without-fix: 126 requests, 1 failed
16:20:36 000

with-fix: 156 requests, 1 failed
16:22:16 000

And each failure next to its sync, from argocd app history, converted to the probe's clock (UTC):

Read that carefully, because it doesn't say "fixed". Without the hook, and while putting it back in (when the old pods being stopped didn't have it yet), there was a failure 2 seconds after the sync: right when the first old pod was being stopped. That's the race, and with the hook it didn't happen. But the rollout with the hook still had one failure, 29 seconds later, and the middle step had a second one at 23 seconds. Those don't line up with pods being stopped, and this measurement doesn't say what caused them. On a cluster running in Docker on a laptop, a DNS lookup or a connection that failed for its own reasons is a fair suspect, but that's a guess, not a finding.

So the honest conclusion: zero downtime isn't automatic, and isn't proven by watching. The shutdown race is real, a preStop pause addresses it, and measuring is how you find out what's left. Real setups measure the same way, with far more requests, and add retries in the clients, so a single dropped request is retried instead of shown to a customer.

Spot the problem

Work out what's wrong before you read the explanation.

The hotfix that undid itself. The bookshop's page has a typo in production, and the fix in Git is waiting for review. Someone with cluster access runs helm upgrade bookshop charts/bookshop -n bookshop --set shop.headline="Books for people who run things." from their laptop, to fix it now. It works, for about three minutes; then the typo is back. They run it again; same thing. What's happening? Argo CD owns the bookshop, and Helm doesn't know that. helm upgrade applied the fixed YAML, and created a Helm release that Argo CD knows nothing about. Then Argo CD's next check saw the cluster no longer matching Git, still with the typo, and self-heal put Git's version back, exactly as it did with kubectl scale in Lesson 1. The release is now a leftover, with Secrets in the namespace, that nobody manages. The fix is to merge the change to Git; to clean up, helm uninstall would delete the resources Argo CD manages too, so delete the release's Secrets instead, and let Argo CD resync.

Where you are now

The bookshop runs from its Helm chart, managed by Argo CD, with no Helm release and its history in Git. A settings change now rolls the pods, so users see it as soon as the rollout ends. And you've measured a rollout, found it dropping the odd request, fixed the cause you could identify, and been honest about the one you couldn't. Lesson 7 gives the chart a second values file for production, and promotes a new version of the shop by changing one value.

Check yourself

Key terms

• valueFiles: the values files an Application gives a chart, in order.
• releaseName: what .Release.Name is when Argo CD renders a chart.
• kubectl diff: a comparison of files with what's running in the cluster, changing nothing.
• Rolling update: replacing a Deployment's pods a few at a time, starting new ones before stopping old ones.
• Checksum annotation: a fingerprint of a ConfigMap in the pod template, so changing the ConfigMap rolls the pods.
• preStop hook: a command a container runs when told to stop, before it actually stops.
• Probe: here, a small pod that requests a page over and over, to measure whether a rollout drops requests.
• Graceful shutdown: finishing the requests already in progress before a program exits.

Go deeper

• The Argo CD documentation: Helm (opens in a new tab). The Helm documentation: Automatically roll deployments (opens in a new tab).
• The Kubernetes documentation: Termination of pods (opens in a new tab).
• Try: change the probe to print curl's exit code too, with -w "%{http_code} %{exitcode}", and run a few rollouts. A 6 is a failed DNS lookup, 7 a refused connection, and 28 a timeout. Which do you see, and when?
• Try: raise replicas to 4 in values.yaml, and measure a rollout again. Do more copies change the number of failures?