Configuration and Secrets

By the end you'll know

Why settings don't belong inside your container image
What ConfigMaps and Secrets are, and when to use each
Two ways a pod can read them: environment variables and files
What happens when you change a setting, and why Secrets aren't as secret as they sound

The problem: one image, many places

The same app usually runs in more than one place: on your laptop, in a test cluster, in production. The code is the same everywhere, but the settings aren't. Each place has its own database address, its own log level, its own passwords.

You could bake the settings into the image, but then you'd need a different image for every place, and anyone who can download the image can read the passwords inside it. Instead, the image stays the same everywhere, and the cluster hands the app its settings when it starts. Kubernetes has two objects for that:

ConfigMap: ordinary settings, like a log level, a feature switch or the address of another service.
Secret: sensitive values, like passwords, API keys and tokens.

They work almost the same way. The difference is how Kubernetes treats them: Secrets are kept separate so you can restrict who may read them, and tools take more care not to print them.

Think of a hotel room. Every room is built the same: that's your image. When you check in, the front desk hands you a welcome card with the Wi-Fi name and breakfast hours (a ConfigMap) and a key card for your door (a Secret). The room doesn't change between guests; what you're handed at the desk does.

A ConfigMap and a Secret hold the settings. The pod reads them as environment variables, as files, or both.

Try it: create a ConfigMap and a Secret

Start a cluster:

BASH

kind create cluster --name learn

Create a ConfigMap with two settings. --from-literal means "here's the value, right on the command line":

BASH

kubectl create configmap web-settings --from-literal=GREETING="hello from a ConfigMap" --from-literal=COLOR=blue

Now a Secret. Typing a real password into a command leaves it in your shell history, where anyone who reads that history can find it. A safer habit is to have the shell ask for it without showing what you type:

BASH

read -s API_KEY

Type anything (it's a practice key) and press Enter. Nothing appears on screen, and that's the point. Then create the Secret from it, and clear the variable:

BASH

kubectl create secret generic web-secret --from-literal=API_KEY="$API_KEY"
unset API_KEY

generic is the everyday kind of Secret, for any key and value. Look at both:

BASH

kubectl get configmap web-settings -o yaml
kubectl get secret web-secret -o yaml

The ConfigMap shows its values in plain text. The Secret shows API_KEY as a jumble of letters. That looks protected, but it isn't. It's base64, a way of writing any data as plain letters and numbers, and anyone can turn it back:

BASH

kubectl get secret web-secret -o jsonpath='{.data.API_KEY}' | base64 -d

Your practice key, in plain text. That's the most important thing to know about Secrets: they're encoded, not encrypted. What actually protects a Secret is who's allowed to read it, which is access control (Lesson 10). Anyone who can read the Secret can read the value.

Try it: give the settings to an app

Save this as app.yaml. It's a Deployment (Lesson 5) whose container reads the ConfigMap and the Secret in both ways:

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: settings-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: settings-demo
  template:
    metadata:
      labels:
        app: settings-demo
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["sleep", "3600"]   # just stay running, so we can look inside
          envFrom:
            - configMapRef:
                name: web-settings     # every key becomes an environment variable
          env:
            - name: API_KEY
              valueFrom:
                secretKeyRef:          # one key from a Secret
                  name: web-secret
                  key: API_KEY
          volumeMounts:
            - name: settings
              mountPath: /etc/settings # the ConfigMap's keys appear here as files
      volumes:
        - name: settings
          configMap:
            name: web-settings

BASH

kubectl apply -f app.yaml
kubectl get pods

When the pod is Running, look inside it. kubectl exec runs a command in a container, and deploy/settings-demo picks a pod from the Deployment for you:

BASH

kubectl exec deploy/settings-demo -- env
kubectl exec deploy/settings-demo -- ls /etc/settings
kubectl exec deploy/settings-demo -- cat /etc/settings/COLOR

env lists GREETING, COLOR and API_KEY among the environment variables. /etc/settings holds one file per key, named after the key, with the value inside. Same settings, two ways in:

Environment variables are the simplest, and what most apps expect.
Files suit apps that read a whole config file, and they have one more advantage, which you're about to see.

Change a setting

Change COLOR to green. kubectl patch changes just the part of an object you name:

BASH

kubectl patch configmap web-settings -p '{"data":{"COLOR":"green"}}'

Now check both places:

BASH

kubectl exec deploy/settings-demo -- printenv COLOR
kubectl exec deploy/settings-demo -- cat /etc/settings/COLOR

The environment variable still says blue. Environment variables are set once, when the container starts, and never change after that. The file catches up by itself: run the cat again over the next minute or two, and it switches to green.

To give the app the new environment variable, restart its pods. rollout restart replaces them one by one, the same way an update does:

BASH

kubectl rollout restart deployment/settings-demo
kubectl exec deploy/settings-demo -- printenv COLOR

green. Remember this one: changing a ConfigMap or Secret doesn't change environment variables in running pods. If you update a setting and the app doesn't notice, it usually needs a restart.

In my real cluster

My API gets five settings when it starts. Four are sensitive and come from Secrets: the database address with its password, an admin token, and two Supabase values for checking users' logins. The fifth, the address of my Redis cache, isn't secret, so it's a plain value right in the Deployment. Each sensitive one is a secretKeyRef, exactly like API_KEY above. My Cloudflare connector works the same way: its tunnel token is a Secret called cloudflared-token.

Everything else about my cluster lives in Git, and Argo CD applies it. Secrets are the exception. Git only holds the references, "use the key token from the Secret cloudflared-token", and never the values. I created each Secret by hand, typing the value with read -s the way you did. Even base64 in Git would be the same as publishing the password.

The trade-off is that those Secrets exist only in the cluster. If I rebuilt it from Git, everything would come back except them, and I'd recreate them by hand. Tools that keep Secrets encrypted in Git, or fetch them from a password vault, solve that. They're a topic for running Kubernetes in production.

My dashboard, PodYard, which you'll meet in Lesson 10, can read almost everything in the cluster, but not Secrets. That's deliberate: a Secret is only as safe as the list of who can read it.

Break it on purpose

Ask for a key that doesn't exist. In app.yaml, change the Secret's key: API_KEY to key: API_KEYY, and apply:

BASH

kubectl apply -f app.yaml
kubectl get pods

The new pod is stuck at CreateContainerConfigError. Kubernetes won't start a container when a setting it asks for is missing, because an app started without its database password would only fail in a more confusing way later. Ask why:

BASH

kubectl get events --field-selector reason=Failed

couldn't find key API_KEYY in Secret default/web-secret. Notice the old pod is still running: as in Lesson 6, the update never finished, so nothing was replaced. When a pod shows CreateContainerConfigError, check the names: the ConfigMap or Secret, and the key inside it. Fix the key back to API_KEY and apply.

Clean up

BASH

kind delete cluster --name learn

Check yourself

Your app runs on your laptop and in production. Each place has a different database password. Where should the password go?
Which belongs in a Secret rather than a ConfigMap?
kubectl get secret shows API_KEY as a string of random-looking letters. Is the value safe?
You change a value in a ConfigMap. A running pod reads it as an environment variable. What does the pod see?
A new pod is stuck at CreateContainerConfigError. What's the most likely cause?
Your whole cluster setup lives in Git. Where should the database password's value go?

Key terms

ConfigMap: an object that holds ordinary settings as keys and values.
Secret: like a ConfigMap, for sensitive values. Access to it can be restricted separately.
base64: a way of writing data as plain letters and numbers. It hides nothing; anyone can decode it.
envFrom: turns every key in a ConfigMap or Secret into an environment variable.
secretKeyRef / configMapKeyRef: sets one environment variable from one key.
CreateContainerConfigError: the pod can't start because a ConfigMap, Secret or key it needs doesn't exist.
kubectl rollout restart: replaces a Deployment's pods one by one, so they pick up new settings.

Go deeper

ConfigMaps (opens in a new tab) in the official docs.
Secrets (opens in a new tab) in the official docs, including the warning about base64 and how to protect Secrets properly.
Good practices for Kubernetes Secrets (opens in a new tab) covers encryption, access control and keeping Secrets out of logs.