Secrets

A secret has many copies

A secret is anything that grants access by being known: a password, an API key, a token, a private key, a webhook address. The trouble is that each one ends up in more places than you meant it to:

• Git, when someone commits it. History keeps it forever, even after it's deleted.
• Files on laptops, next to the code that uses it.
• The cluster, as a Kubernetes Secret, and in the database behind it.
• Backups of all of the above.
• Chat, tickets, logs and screenshots. (This series has had two: a Slack webhook pasted into a chat, and a screenshot with ping addresses in it.)

Protecting a secret means knowing every copy, and when one leaks, replacing the secret itself rather than chasing the copies.

1. The Secrets in the cluster

kubectl get secrets -A --no-headers -o custom-columns=TYPE:.type | sort | uniq -c | sort -rn

  16 Opaque
   2 kubernetes.io/tls
   1 kubernetes.io/service-account-token

Nineteen, the same kinds Lesson 2 listed by name: the Argo CD and Grafana logins, the Slack and heartbeat addresses, the Sealed Secrets key, and the generated settings and certificates the charts make.

2. Secrets in Git

gitleaks is a free scanner that looks for secrets by pattern: a few hundred rules for known formats (AWS keys, GitHub tokens, Slack webhooks, private keys) and for long random-looking strings. Run from its container image, over the repository's whole history:

docker run --rm -v "$PWD:/repo:ro" zricethezav/gitleaks:v8.18.4 detect --source /repo --redact --no-banner

• detect --source /repo scans every commit in the repository's history, not just the files as they are now.
• --redact never prints what it finds.
• :ro mounts the folder read-only.

8:34PM INF scan completed in 175ms
8:34PM INF no leaks found

Clean. This is the scan that matters most: the repository is public, so its history is what anyone in the world can read. Every secret in it went in as a SealedSecret, or was kept out by .gitignore.

3. Secrets on disk

The same scan with --no-git reads the folder's files as they are, including the ones Git ignores:

8:34PM WRN leaks found: 1

1  slack-webhook-url  .slack-webhook

One. But the folder holds six secret files, and Git is ignoring all of them:

$ git check-ignore -v .argocd-admin .grafana-admin .slack-webhook .healthchecks-ping .healthchecks-api .sealed-secrets-key.yaml
.argocd-admin
.grafana-admin
.slack-webhook
.healthchecks-ping
.healthchecks-api
.sealed-secrets-key.yaml

The scanner found one in six. A Slack webhook has a format it knows (hooks.slack.com/services/...). The two passwords are random strings without a recognisable format. The Healthchecks ping addresses are ordinary-looking web addresses. And the Sealed Secrets private key, the most valuable of the six, is stored base64-encoded inside YAML, so it doesn't look like a private key either.

A clean scan means "nothing matched a rule", not "no secrets". Scanners are a net for mistakes; they don't replace knowing where your secrets are.

4. A Secret, straight from etcd

A demo Secret, and the same Secret three ways:

kubectl create secret generic lesson5-demo -n default --from-literal=password=correct-horse-battery-staple
kubectl get secret lesson5-demo -n default -o jsonpath='{.data.password}'

Y29ycmVjdC1ob3JzZS1iYXR0ZXJ5LXN0YXBsZQ==

That's base64, an encoding that turns any bytes into letters and digits so they fit in YAML. It isn't encryption: there's no key, and anyone can reverse it.

$ kubectl get secret lesson5-demo -n default -o jsonpath='{.data.password}' | base64 -d
correct-horse-battery-staple

Then the copy that matters: etcd, the database where the API server keeps everything. etcdctl is its command-line client, run inside the etcd pod with etcd's own certificate:

kubectl -n kube-system exec etcd-gitops-control-plane -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/lesson5-demo --print-value-only

• /registry/secrets/NAMESPACE/NAME is where the API server keeps each Secret.
• --print-value-only prints just the stored bytes. (The script turned anything unprintable into dots.)

k8s.....v1..Secret........lesson5-demo....default".*$7a69ec02-b608-4d15-b12a-f4139585bd372.8.B...........e..kubectl-create..Update..v1".........2.FieldsV1:1./{"f:data":{".":{},"f:password":{}},"f:type":{}}B..(..password.

The Secret, as plain bytes: its name, namespace and fields, readable as they are. The value comes a few bytes later, after password, not even base64-encoded: etcd stores the raw bytes. (Section 7 finds it in etcd's files.) Anyone who gets etcd's files (a backup, a copy of a node's disk, a snapshot of the volume) gets every Secret in the cluster, without ever talking to the API server.

5. Encryption at rest

Encryption at rest means the API server encrypts each Secret before writing it to etcd, and decrypts it when reading it back. It's off by default. Turning it on takes two things.

A key, and a configuration file. Written straight into the control-plane node, never printed:

$ docker exec gitops-control-plane ls -l /etc/kubernetes/enc/
-rw------- 1 root root 268 Nov  7 20:34 encryption.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - secretbox:
          keys:
            - name: key1
              secret: (32 random bytes, base64)
      - identity: {}

• resources: [secrets]: encrypt Secrets. (ConfigMaps and other kinds can be added; Secrets are the usual start.)
• secretbox: the encryption method, XSalsa20 with Poly1305, which both hides the data and detects any change to it. The Kubernetes documentation recommends it over the older aescbc.
• The first provider is used for writing; all of them are tried for reading. identity: {}, meaning "not encrypted", comes last so Secrets written before the change can still be read.
• key1 is the key's name. To change keys later, you add key2 above it, rewrite every Secret, then remove key1.

A setting on the API server. On kind, as on any cluster built with kubeadm, the API server is a static pod: the kubelet runs whatever manifest is in /etc/kubernetes/manifests/, and restarts it when the file changes. The change: one flag, and a mount so the API server can read the file:

$ docker exec gitops-control-plane diff /root/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml
14a15
>     - --encryption-provider-config=/etc/kubernetes/enc/encryption.yaml
82a84,86
>     - mountPath: /etc/kubernetes/enc
>       name: encryption-config
>       readOnly: true
104a109,112
>   - hostPath:
>       path: /etc/kubernetes/enc
>       type: DirectoryOrCreate
>     name: encryption-config

The kubelet sees the changed file and restarts the API server. Waiting for it...
API server ready again after about 46 seconds

• The backup went to /root, not the manifests folder. The kubelet runs every file in manifests/, so a backup copy there would start a second API server.
• 46 seconds without an API server. Running pods carried on; nothing new could be scheduled, and kubectl couldn't connect. On a real cluster with three control-plane nodes, you change one at a time, and the API stays up.

Afterwards, two pods were briefly restarting; a minute later, nothing was left in a state other than Running or Completed.

6. What's encrypted now?

The demo Secret, read straight from etcd again:

k8s.....v1..Secret........lesson5-demo....default".*$7a69ec02-b608-4d15-b12a-f4139585bd372.8.B......

Unchanged. Encryption happens when a Secret is written, and this one was written before. So every Secret gets rewritten, unchanged:

kubectl get secrets -A -o json | kubectl replace -f -

  20 replaced

(Twenty: the nineteen, and the demo.) The demo Secret now:

k8s:enc:secretbox:v1:key1:8%..|T..Yg.d....-..._.ET7...`69k..N.S......r.bi..:.,c.TT.T....){h...iZ....

A label saying how it was encrypted, and with which key, then nothing readable. And every Secret in etcd, checked the same way:

  20 encrypted (secretbox, key1)

All twenty. Then, through the API:

$ kubectl get secret lesson5-demo -n default -o jsonpath='{.data.password}' | base64 -d
correct-horse-battery-staple

Nothing changed for anyone who uses the API, which is the point and the limit. Encryption at rest protects Secrets from someone with etcd's files. It does nothing about someone who can read Secrets through the API, like the log collector from Lesson 1 that can read every Secret in the cluster. That's least privilege, Lesson 9.

Where the key lives

$ docker exec gitops-control-plane ls -l /etc/kubernetes/enc/encryption.yaml /var/lib/etcd/member/snap/db
-rw------- 1 root root      268 Nov  7 20:34 /etc/kubernetes/enc/encryption.yaml
-rw------- 1 root root 34680832 Nov  7 20:35 /var/lib/etcd/member/snap/db

On the same machine as the data it protects. A copy of this node's disk has both. That's still worth having: etcd backups usually go somewhere else, without the key. But it's why real clusters use a KMS provider instead: the API server asks a key management service (AWS KMS, Google Cloud KMS, Azure Key Vault, HashiCorp Vault) to encrypt a small data key, and the main key never leaves that service. Managed clusters do this for you: EKS, GKE and AKS encrypt etcd, and let you bring your own KMS key.

7. The old copies

The demo Secret was encrypted, then deleted. So was it gone? A search of etcd's files, passing each password in on input and printing only which files contain it:

$ docker exec gitops-control-plane ls -l /var/lib/etcd/member/snap /var/lib/etcd/member/wal
/var/lib/etcd/member/snap:
-rw-r--r-- 1 root root    7150 Nov  7 05:32 0000000000000002-000000000001adbd.snap
...
-rw------- 1 root root 8388608 Nov  7 20:37 db

/var/lib/etcd/member/wal:
-rw------- 1 root root 64005336 Nov  5 18:47 0000000000000000-0000000000000000.wal
-rw------- 1 root root 64004832 Nov  6 01:52 0000000000000001-000000000000b859.wal
-rw------- 1 root root 64008496 Nov  7 16:46 0000000000000002-0000000000014a17.wal
-rw------- 1 root root 64000000 Nov  7 20:37 0000000000000003-00000000000203e4.wal
-rw------- 1 root root 64000000 Nov  7 16:46 1.tmp

Files holding the demo password:
/var/lib/etcd/member/snap/db
/var/lib/etcd/member/wal/0000000000000003-00000000000203e4.wal
Files holding Grafana's admin password:
/var/lib/etcd/member/snap/db
/var/lib/etcd/member/wal/0000000000000000-0000000000000000.wal

Both, in plain text, in two places. etcd keeps two kinds of file:

• db, the database. etcd keeps every version of every key, not just the latest, until it compacts its history (the API server asks it to every 5 minutes). Even then, the space isn't cleared until it's defragmented, which rewrites the file.
• The write-ahead log (WAL). Every change is written here first, so etcd can recover after a crash. It keeps the last five files, 64 MB each, and deletes the oldest only when a new one starts. Grafana's password is in the very first one, from November 5: the day the cluster was built.

A defragment on its own wasn't enough (the file shrank from 18 MB to 7.4 MB, and both passwords were still there). Compacting the history up to now, then defragmenting:

$ etcdctl compact 136644 --physical
compacted revision 136644

$ etcdctl defrag
Finished defragmenting etcd member[127.0.0.1:2379]. took 42.324666ms

Files holding the demo password:
/var/lib/etcd/member/wal/0000000000000003-00000000000203e4.wal
Files holding Grafana's admin password:
/var/lib/etcd/member/wal/0000000000000000-0000000000000000.wal

The database is clean; the write-ahead log isn't, and nothing short of time clears it: those files go when enough new changes push them out. And any etcd backup taken before today has every Secret in plain text, for as long as the backup is kept.

The lesson real teams draw from this: turn on encryption at rest when the cluster is built, and if you turn it on later, change the secrets themselves afterwards. Encrypting a Secret protects the next copy, not the ones already written.

8. The Sealed Secrets key

Lesson 2 called it the crown jewel: the private key that decrypts every SealedSecret in the public repository's history.

$ kubectl -n kube-system get secrets -l sealedsecrets.bitnami.com/sealed-secrets-key
NAME                      CREATED                STATUS
sealed-secrets-keygzzt8   2026-11-04T23:14:44Z   active

$ kubectl -n kube-system get deploy -l app.kubernetes.io/name=sealed-secrets -o jsonpath='{.items[0].spec.template.spec.containers[0].args}'
["--update-status","--key-prefix","sealed-secrets-key","--listen-addr",":8080","--listen-metrics-addr",":8081"]

• One key, made November 4. With no --key-renew-period set, the controller makes a new key every 30 days and keeps the old ones, so it can still unseal older SealedSecrets. New seals use the newest key.
• A backup that holds one key stops being enough on day 31. If the cluster were rebuilt after that, SealedSecrets made with the newer key couldn't be unsealed.

The backup in the repository folder, checked against the cluster (comparing a hash of each certificate):

  sealed-secrets-keygzzt8   backup cbad6485c000   cluster cbad6485c000

It matches, today. So the script made a fresh backup of every key, by label, outside the repository folder:

mkdir -p ~/.config/gitops-practice && chmod 700 ~/.config/gitops-practice
(umask 077; kubectl -n kube-system get secrets -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > ~/.config/gitops-practice/sealed-secrets-keys.yaml)

• -l sealedsecrets.bitnami.com/sealed-secrets-key selects every key the controller has, old and new.
• umask 077 makes the file readable only by you from the moment it's created.

-rw-------  1 mayowababatola  staff  14195 Nov  7 15:35 /Users/mayowababatola/.config/gitops-practice/sealed-secrets-keys.yaml
1

Out of the repository folder means one wrong git add -f can no longer publish it. The better home is a password manager, or a file encrypted with age or gpg, kept somewhere other than this laptop. Re-run the backup after each new key appears, monthly.

Spot the problem

Work out what's wrong before you read the explanation.

The audit that passed, and the backup that didn't. An auditor asks whether a team's Kubernetes Secrets are encrypted at rest. The team turns it on, rewrites every Secret, shows the auditor k8s:enc:secretbox on every key in etcd, and the finding is closed. Three months later, an old etcd backup from a storage bucket turns up in a breach. What did it contain? Every Secret, in plain text. Encryption at rest protects copies written after it's turned on. Everything written before (backups, older versions of each key in the database until it's compacted, and the write-ahead log, which in this lesson still held a password from the day the cluster was built) stays as it was. Rewriting the Secrets made them encrypted; it didn't make the old copies go away. Three fixes: change the secrets themselves after turning encryption on, so the old copies hold values that no longer work; delete or re-encrypt old etcd backups, and encrypt the backups as well; and turn encryption on when a cluster is built, so there's never a plain copy to start with.

Where you are now

You know where the practice setup's secrets are, what each copy is protected by, and what a scanner can and can't find. Secrets are encrypted in etcd now, and you know why the old copies outlive the change. The next lesson looks at what runs those Secrets: the container images and charts the cluster pulls from six registries and five chart repositories, and what's known to be wrong with them.

Check yourself

Key terms

• Secret: anything that grants access by being known. Also Kubernetes's object for holding one.
• base64: a reversible encoding of bytes as text. Not encryption.
• etcd: the database where the Kubernetes API server keeps everything.
• Encryption at rest: encrypting data where it's stored, here Secrets in etcd.
• EncryptionConfiguration: the file telling the API server which resources to encrypt, how, and with which keys.
• KMS provider: encryption at rest with the main key kept in a key management service, not on the node.
• Static pod: a pod the kubelet runs from a file in its manifests folder, as kubeadm does for the API server.
• Compaction and defragmentation: dropping etcd's old versions of each key, and rewriting its file to free the space.
• Write-ahead log (WAL): the record of every change etcd keeps for crash recovery, separate from its database.
• gitleaks: a scanner that finds secrets in Git history and files, by pattern.

Go deeper

• Kubernetes: Encrypting confidential data at rest (opens in a new tab), and using a KMS provider (opens in a new tab).
• gitleaks (opens in a new tab), and its rules file, to see exactly what it looks for.
• Sealed Secrets (opens in a new tab): the sections on key renewal and backup.
• OWASP's Secrets Management Cheat Sheet (opens in a new tab).
• Try: add --key-renew-period=0 to the Sealed Secrets controller's arguments. What does that turn off, and what would you have to do instead?
• Try: add a gitleaks custom rule that matches your Healthchecks ping addresses, and run the folder scan again.