Secrets in GitOps

The one thing that can't go in Git

Everything the bookshop runs is now in Git: the chart, each environment's values, even the Applications. The next thing a real shop needs is a password, for a database, say. Kubernetes keeps those in a Secret:

kubectl create secret generic bookshop-db -n bookshop \
  --from-literal=password=correct-horse-battery --dry-run=client -o yaml

apiVersion: v1
data:
  password: Y29ycmVjdC1ob3JzZS1iYXR0ZXJ5
kind: Secret
metadata:
  creationTimestamp: null
  name: bookshop-db
  namespace: bookshop

--dry-run=client -o yaml only prints it; nothing is created. The password looks scrambled, but it isn't:

echo Y29ycmVjdC1ob3JzZS1iYXR0ZXJ5 | base64 -d

correct-horse-battery

Base64 is encoding, not encryption. It turns any bytes into plain letters, so they survive YAML, and anyone can turn them back. Committing this file would put the password in Git for good: deleting it later leaves it in the history, on every clone, readable by everyone who can read the repository. GitOps needs a way to put the description of a secret in Git without the secret itself.

How Sealed Secrets works

Sealed Secrets splits the job between two pieces:

• A controller in the cluster, which makes a key pair when it first starts: a public key that anyone can have, and a private key that never leaves the cluster.
• kubeseal, a command on your computer that fetches the public key and uses it to encrypt a value.

What you commit is a SealedSecret: a new kind of resource holding the encrypted value. Argo CD applies it like anything else. The controller sees it, decrypts it with the private key, and makes an ordinary Secret from it. Anything can lock; only the cluster can unlock.

Installing it with Argo CD

The controller comes as a Helm chart, published by the Sealed Secrets project. Helm charts can come from a chart repository, a web address that lists charts and their versions:

helm repo add sealed-secrets https://bitnami.github.io/sealed-secrets
helm search repo sealed-secrets/sealed-secrets

NAME                         	CHART VERSION	APP VERSION	DESCRIPTION
sealed-secrets/sealed-secrets	2.22.0       	0.41.2     	Helm chart for the sealed-secrets controller.

Two versions, as in Lesson 5: chart 2.22.0, which packages Sealed Secrets 0.41.2. To install it the GitOps way, one more file in argocd/, for the root to pick up:

# Sealed Secrets (Lesson 10): a controller that turns the encrypted SealedSecrets
# in this repository into ordinary Secrets in the cluster. From its own Helm chart.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://bitnami.github.io/sealed-secrets
    chart: sealed-secrets
    targetRevision: 2.22.0
    helm:
      releaseName: sealed-secrets
      valuesObject:
        # The name kubeseal looks for by default.
        fullnameOverride: sealed-secrets-controller
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The source is different from the bookshop's. Instead of a Git path, it names a chart from a chart repository, and targetRevision is the chart version, pinned, so the controller only changes when this file does. valuesObject sets the chart's values right in the Application, here just a name. Merged, and a moment later:

deployment "sealed-secrets-controller" successfully rolled out

$ status sealed-secrets
Synced Healthy 2.22.0

$ kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets
NAME                                        READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-8b68c74c7-4ccq4   1/1     Running   0          6s

The revision Argo CD reports is 2.22.0, the chart version, where the bookshop's is a commit. A pull request, and a new tool in the cluster, installed the same way as the shop.

Sealing a password

kubeseal fetches the public key from the controller:

kubeseal --version
kubeseal --fetch-cert | head -3

kubeseal version: v0.31.0
-----BEGIN CERTIFICATE-----
MIIEzDCCArSgAwIBAgIQHG0U0QX7FZ1VnoX3qqx15jANBgkqhkiG9w0BAQsFADAA
MB4XDTI2MTEwNDIyMzc0MloXDTM2MTEwMTIyMzc0MlowADCCAiIwDQYJKoZIhvcN

A certificate, holding the public key: safe to share, safe to commit. Then a password, made just for this lesson, and sealed:

PW=$(openssl rand -hex 12)
ENC=$(printf %s "$PW" | kubeseal --raw --namespace bookshop --name bookshop-db)
echo "${ENC:0:64}... (${#ENC} characters)"

the password (a practice one, made just now): f994119e3a31e48126d04262

AgBm7k4wU43dPHni5hkTd96Q4MzYKo5qY7iMmDNcrZf7/Q86DH96342ahKr/lj2S... (740 characters)

--raw seals one value, rather than a whole Secret file. --namespace and --name say which Secret it's for, and they're sealed in with it: more on that below. A 24-character password became 740 characters. This one is base64 too, but of encrypted bytes. Decoding it gets you this:

00000000: 0200 66ee 4e30 538d dd3c 79e2 e619 1377  ..f.N0S..<y....w
00000010: de90 e0cc d82a 8e6a 63b8 8c98 335c ad97  .....*.jc...3\..
00000020: fbfd 0f3a 0c7f 7adf 8d9a 84aa ff96 3d92  ...:..z.......=.

Noise. Compare correct-horse-battery.

The chart, with a password

The chart gets one new template, templates/sealedsecret.yaml, which makes a SealedSecret only when an environment has an encrypted value:

{{- if .Values.db.encryptedPassword }}
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: {{ .Release.Name }}-db
  labels:
    {{- include "bookshop.labels" . | nindent 4 }}
spec:
  encryptedData:
    password: {{ .Values.db.encryptedPassword }}
  template:
    metadata:
      name: {{ .Release.Name }}-db
{{- end }}

The Deployment hands the Secret to nginx as an environment variable, under the same condition:

          {{- if .Values.db.encryptedPassword }}
          env:
            # From the Secret the Sealed Secrets controller makes (Lesson 10).
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ .Release.Name }}-db
                  key: password
          {{- end }}

nginx has no use for a database password, of course. It stands in for an app that would. The default in values.yaml is empty, and production's file gets the sealed value:

 db:
-  encryptedPassword: ""
+  encryptedPassword: "AgBm7k4wU43dPHni5hkTd96Q4MzYKo5qY7iMmDNcrZf7/Q86DH96342ahKr/lj2Sa6a

Staging has no value, so the check from Lesson 7 applies: does staging's render change?

helm template bookshop-staging charts/bookshop -f charts/bookshop/values.yaml \
  -f charts/bookshop/values-staging.yaml | grep -c -e SealedSecret -e DB_PASSWORD

0

Nothing for staging. Merged, and production:

deployment "bookshop" successfully rolled out
Synced Healthy dd5a1f40802fc06

$ kubectl get sealedsecret,secret -n bookshop
NAME                                   STATUS   SYNCED   AGE
sealedsecret.bitnami.com/bookshop-db            True     6s

NAME                 TYPE     DATA   AGE
secret/bookshop-db   Opaque   1      6s

$ kubectl get events -n bookshop --field-selector involvedObject.kind=SealedSecret
REASON     MESSAGE
Unsealed   SealedSecret unsealed successfully

Argo CD applied the SealedSecret; the controller unsealed it and made secret/bookshop-db from it, in the same few seconds. The new environment variable changed the pod template, so the pods rolled: new ReplicaSet 74d9f67bc8, as in every release.

What the cluster has, and what Git has

The cluster has the real password, in the Secret and in the pod:

kubectl get secret bookshop-db -n bookshop -o jsonpath='{.data.password}' | base64 -d; echo
kubectl exec -n bookshop deploy/bookshop -- printenv DB_PASSWORD

f994119e3a31e48126d04262
f994119e3a31e48126d04262

And Git? Search the values file, then every commit in the repository's history:

grep -c 'f994119e3a31e48126d04262' charts/bookshop/values-production.yaml
git log -p | grep -c 'f994119e3a31e48126d04262'

0
0

Zero, and zero. The password reached the pod without ever being written to Git. What's in Git is the 740 characters only this cluster can read.

Notice what that first command did, though. Anyone who can read Secrets in the bookshop namespace can read the password, exactly as easily as before. Sealed Secrets protects the password in Git. In the cluster, it's an ordinary Secret, and who can read it is up to Kubernetes permissions.

Sealed for one place

The sealed value said --namespace bookshop --name bookshop-db. What if someone copies it somewhere else? A throwaway namespace, scratch, and the same encrypted value in a SealedSecret there:

$ kubectl apply -f scratch-sealed.yaml
sealedsecret.bitnami.com/bookshop-db created

$ kubectl get secret bookshop-db -n scratch
Error from server (NotFound): secrets "bookshop-db" not found

$ kubectl get events -n scratch --field-selector involvedObject.kind=SealedSecret
REASON            MESSAGE
ErrUnsealFailed   Failed to unseal: no key could decrypt secret (password)

Refused. The namespace and name are part of what was encrypted, so the same value can't be unsealed anywhere else. That's the point: anyone allowed to create things in some other namespace can't copy production's sealed password there and read the result. It's also why staging, when it needs a password, gets its own, sealed with --namespace bookshop-staging. The scratch namespace was then deleted.

The key that unlocks everything

Everything sealed so far depends on one thing: the controller's private key.

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

NAME                      TYPE                DATA   AGE
sealed-secrets-keygzzt8   kubernetes.io/tls   2      33s

The key pair, as a Secret, made when the controller first started. If the cluster is lost, so is this, and every sealed value in Git becomes 740 characters nobody can read. A new controller makes a new key, which can't decrypt the old values. So the key gets a backup:

echo '.sealed-secrets-key.yaml' >> .gitignore
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > .sealed-secrets-key.yaml
chmod 600 .sealed-secrets-key.yaml
ls -l .sealed-secrets-key.yaml

-rw-------  1 mayowababatola  staff  7085 Nov  4 17:38 .sealed-secrets-key.yaml

The backup goes in .gitignore before the file exists, so it can't be committed by accident, and only you can read it, like .argocd-admin from Lesson 2. The one file that must never go in Git is the one that unlocks everything that does. Lesson 11 needs it.

Spot the problem

Work out what's wrong before you read the explanation.

The version that wasn't there. This happened while making this lesson. A script looked up the chart's latest version with helm search repo, wrote it into argocd/sealed-secrets.yaml, and merged the file. The chart repository's address had moved, so helm repo add failed with a 404, the search found nothing, and the script carried on. What got merged said targetRevision: null. Then kubeseal failed, because there was no controller to fetch a key from, and the chart change after it merged with an empty password. The script merged both pull requests itself, so nobody looked. What went wrong? The script didn't stop on the first failure. A failed lookup became an empty value, and an empty value became a commit. It's still in this repository's history, as #34 and #35, followed by #37 and #38, which set the real version and value. The fix was two lines: check the version isn't empty before writing it, and stop if kubeseal returns nothing. Anything that writes to Git automatically should treat "I got nothing" as an error, not as a value.

Where you are now

The bookshop has a password, and Git describes it without containing it. You've installed a tool from someone else's Helm chart the GitOps way, and you've got two files that must never go in Git: .argocd-admin and .sealed-secrets-key.yaml. Lesson 11 puts everything to the test: delete the whole cluster, and rebuild it from Git, with that key, as the only thing you bring.

Check yourself

Key terms

• Secret: a Kubernetes resource for sensitive values, stored base64 encoded, not encrypted.
• Base64: a way of writing any bytes as plain letters. Anyone can decode it.
• Sealed Secrets: a controller and a command, kubeseal, for keeping encrypted secrets in Git.
• SealedSecret: the resource holding an encrypted value, which the controller turns into a Secret.
• Public key, private key: a pair. What one locks, only the other can unlock.
• kubeseal: the command that encrypts values with the cluster's public key.
• Chart repository: a web address listing Helm charts and their versions.
• External Secrets Operator: a controller that copies secrets from a secrets store into the cluster, instead of keeping them in Git.

Go deeper

• The Sealed Secrets README (opens in a new tab), especially scopes and key renewal, and the External Secrets Operator (opens in a new tab).
• Try: give staging its own password, sealed with --namespace bookshop-staging --name bookshop-staging-db, in values-staging.yaml. Which pods roll, and does production change?
• Try: change the password by hand with kubectl edit secret bookshop-db -n bookshop. Does the controller put it back? Does Argo CD notice?