Least privilege in Kubernetes

What least privilege means

Least privilege is giving everything only the access it needs to do its job, and no more. It doesn't stop an attacker getting in; it limits what they get when they do. If the bookshop's nginx is ever compromised, the question is what that nginx could do next, and the answer should be "very little".

In Kubernetes, that's three layers:

1. The container: which user it runs as, which Linux powers it has, what it can write.
2. The service account: what it can ask the Kubernetes API to do.
3. The network: who can connect to it, and who it can connect to.

Lesson 1 found a problem on each. This lesson fixes them for staging, checking each one from inside, from outside, and through the API.

1. Pod Security Standards: what each namespace would fail

Kubernetes has three built-in Pod Security Standards:

• privileged: no limits.
• baseline: blocks the known ways out of a container: host access, privileged mode, extra capabilities.
• restricted: baseline plus current best practice: non-root, no privilege escalation, every capability dropped, a system call filter.

A namespace opts in with a label, and Pod Security Admission, built into the API server, enforces it. Before turning it on, a dry run asks the server what it would reject, without changing anything:

kubectl label --dry-run=server --overwrite ns NAMESPACE pod-security.kubernetes.io/enforce=restricted

--- baseline
  argocd               passes
  bookshop             passes
  bookshop-staging     passes
  ...
  kube-system          4 pods fail
      etcd-gitops-control-plane (and 3 other pods): host namespaces, hostPath volumes, hostPort, probe or lifecycle host
      kindnet-bn9n5 (and 1 other pod): non-default capabilities, host namespaces, hostPath volumes
      kube-proxy-dcng6 (and 1 other pod): host namespaces, hostPath volumes, privileged
  monitoring           2 pods fail
      monitoring-prometheus-node-exporter-sdtsm (and 1 other pod): host namespaces, hostPath volumes, hostPort

--- restricted
  argocd               passes
  bookshop             2 pods fail
      bookshop-65d8d9ff66-2krfn (and 1 other pod): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
  bookshop-staging     1 pods fail
  ...
  monitoring           3 pods fail
      alert-receiver-57b6c5c4b-w8hkl (and 1 other pod): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile

How to read it:

• kube-system and the node exporter fail even baseline, and that's correct. The control plane, the network plugin and a tool that reads each node's own statistics need the host. Those namespaces stay at privileged on purpose, with as little else in them as possible.
• Argo CD passes restricted already. Its charts were built that way.
• The bookshops fail restricted on the same four things: privilege escalation, capabilities, root, and no system call filter. That's what this lesson fixes in staging.

2. Inside the bookshop's nginx, before

$ id
uid=0(root) gid=0(root) groups=0(root)

$ grep -E "^(CapEff|NoNewPrivs|Seccomp):" /proc/1/status
CapEff:	00000000a80425fb
NoNewPrivs:	0
Seccomp:	0

$ touch /etc/nginx/written-by-an-attacker
wrote to /etc/nginx

$ ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt
namespace
token

• uid=0: root inside the container. Not root on the node, but one container escape bug away from it.
• CapEff: the process's effective capabilities, the pieces root's power is split into (changing file ownership, binding low ports, sending raw network packets). a80425fb is the default set container runtimes give every container: fourteen capabilities nginx doesn't need once it has started.
• NoNewPrivs: 0: the process could gain more privileges, through a setuid program for instance.
• Seccomp: 0: no seccomp filter, so every one of the kernel's hundreds of system calls is available. Most container escapes go through a system call ordinary programs never use.
• A writable file system: an attacker could change nginx's configuration or drop tools onto the disk.
• A token for the Kubernetes API, which it never uses. (Its service account can't do anything today. A token nobody needs is still one more thing to steal, and one role change away from mattering.)

3. Who can read every Secret, and why

Every service account in the cluster, asked the same question:

kubectl auth can-i list secrets -A --as=system:serviceaccount:NAMESPACE:NAME

  argocd:argocd-application-controller
  argocd:argocd-applicationset-controller
  cert-manager:cert-manager
  cert-manager:cert-manager-cainjector
  kube-system:generic-garbage-collector
  kube-system:namespace-controller
  kube-system:resourcequota-controller
  kube-system:sealed-secrets-controller
  monitoring:alloy
  monitoring:monitoring-grafana
  monitoring:monitoring-kube-prometheus-operator
  monitoring:monitoring-kube-state-metrics

Twelve. Most need it: Argo CD applies Secrets, cert-manager and Sealed Secrets make them, and the controllers in kube-system are Kubernetes itself. Two stand out, and the roles behind them say why:

--- monitoring:alloy
ClusterRoleBinding alloy -> ClusterRole alloy
    core: pods,pods/log,namespaces [get,list,watch]
    ...
    core: configmaps,secrets [get,list,watch]
    ...

--- monitoring:monitoring-grafana
ClusterRoleBinding monitoring-grafana-clusterrolebinding -> ClusterRole monitoring-grafana-clusterrole
    core: configmaps,secrets [get,watch,list]

grafana-sc-dashboard: METHOD=WATCH  LABEL=grafana_dashboard  RESOURCE=both  NAMESPACE=ALL
grafana-sc-datasources: METHOD=WATCH  LABEL=grafana_datasource  RESOURCE=both

• Alloy's chart grants thirteen rules, enough for every feature Alloy has. This configuration uses one of them: finding pods and reading their logs. It mentions Secrets zero times.
• Grafana's role exists for its sidecars, the small containers that load dashboards and data sources from labelled objects. They're set to RESOURCE=both: ConfigMaps and Secrets, in every namespace. Every dashboard here is a ConfigMap.

That's the usual story: charts grant enough for every option, so the defaults work for everyone. Least privilege means narrowing them to what your setup uses. And since Lesson 8, "every Secret" includes the practice CA's private key.

4. What an ordinary pod can reach

$ kubectl get networkpolicies -A
NAMESPACE   NAME                                              POD-SELECTOR
argocd      argocd-application-controller-network-policy      app.kubernetes.io/name=argocd-application-controller
...
argocd      argocd-server-network-policy                      app.kubernetes.io/name=argocd-server

  http://bookshop.bookshop-staging.svc/                                       200
  http://bookshop.bookshop.svc/                                               200
  http://monitoring-kube-prometheus-alertmanager.monitoring.svc:9093/api/v2/status 200
  http://loki.monitoring.svc:3100/ready                                       200
  http://monitoring-grafana.monitoring.svc/api/health                         200

From a pod in default, everything answers. Kubernetes networking is open by default: any pod can connect to any other, in any namespace. A NetworkPolicy changes that for the pods it selects. Argo CD ships its own; nothing else has one.

5. The change

One pull request, checked before it was opened.

First, do the charts support what this relies on? The script downloaded the exact chart versions the cluster runs and searched them:

kube-prometheus-stack/charts/grafana/values.yaml:26:  # useExistingClusterRole: name-of-some-clusterRole
kube-prometheus-stack/charts/grafana/templates/clusterrolebinding.yaml:18:  {{- if .Values.rbac.useExistingClusterRole }}
kube-prometheus-stack/charts/grafana/templates/clusterrolebinding.yaml:19:  name: {{ .Values.rbac.useExistingClusterRole }}
177:rbac:
178-  # -- Whether to create RBAC resources for Alloy.

Both settings exist. Checking the chart's source beats trusting memory or a blog post: values change between versions.

The bookshop: a hardened option, on in staging

      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 101     # the nginx user in the nginx image
        runAsGroup: 101
        seccompProfile:
          type: RuntimeDefault
        sysctls:
          # Lets a non-root process listen on ports below 1024 (80 and 443), in this pod only.
          - name: net.ipv4.ip_unprivileged_port_start
            value: "0"

• automountServiceAccountToken: false: no API token in the pod.
• runAsNonRoot and runAsUser: 101: run as the image's own nginx user, and refuse to start as root.
• seccompProfile: RuntimeDefault: the container runtime's standard system call filter, which blocks the rarely used, risky calls.
• The sysctl: ports below 1024 have traditionally needed root. This kernel setting, for this pod's network only, lets any user bind them, so nginx keeps listening on 80 and 443 without root or a capability. (The other way is moving nginx to 8080 and 8443 and changing the Service.)

Then, for each container:

          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

And three small writable folders, because nginx does need to write somewhere: its cache, its process ID file, and temporary files. Each is an emptyDir, an empty folder that exists only as long as the pod:

            - name: cache
              mountPath: /var/cache/nginx
            - name: run
              mountPath: /run
            - name: tmp
              mountPath: /tmp

Who may connect: a NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: bookshop
spec:
  podSelector:
    matchLabels:
      app: bookshop
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring

• podSelector: this policy applies to the bookshop's pods.
• policyTypes: [Ingress]: it controls incoming connections. Once a pod is selected by any ingress policy, everything not allowed is refused.
• from the monitoring namespace: Prometheus scrapes the bookshop, and the blackbox exporter probes it. Nothing else needs to. (A real shop would also allow its ingress controller.)

The namespace enforces the restricted standard

In staging's Argo CD Application, so Argo CD labels the namespace:

    managedNamespaceMetadata:
      labels:
        pod-security.kubernetes.io/enforce: restricted
        pod-security.kubernetes.io/warn: restricted

Alloy and Grafana: smaller roles

Alloy's chart is told not to make its own role (rbac.create: false), and a new file in apps/monitoring gives it one with just what reading logs takes:

kind: ClusterRole
metadata:
  name: alloy-read-logs
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "namespaces"]
    verbs: ["get", "list", "watch"]

Grafana's sidecars are told to load from ConfigMaps only (resource: configmap), and its chart is pointed at a new role that can read ConfigMaps and nothing else (rbac.useExistingClusterRole: grafana-read-configmaps).

Checked before committing

Production (should print nothing):
(nothing: production unchanged)

$ helm template ... values-staging.yaml --show-only templates/deployment.yaml | kubectl apply --dry-run=server -n l9-check -f -
deployment.apps/bookshop created (server dry run)
(and production's, for comparison)

$ helm template ... values-production.yaml --show-only templates/deployment.yaml | kubectl apply --dry-run=server -n l9-check -f -
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "nginx", "exporter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "nginx", "exporter" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "nginx", "exporter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "nginx", "exporter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

The API server judged both Deployments against the restricted standard, in a throwaway namespace labelled to warn (created for the check, deleted after). Staging's: no warnings. Production's: every one. The pull request went in as #76.

6. Applying it: three things that went wrong

"Synced" isn't "done"

The first check after the merge said everything was ready, instantly:

  monitoring-config Synced Healthy       after 0s
  alloy             Synced Healthy       after 0s
  monitoring        Synced Healthy       after 0s
  bookshop-staging  Synced Healthy       after 0s

Argo CD hadn't noticed the merge yet, so every app was synced to the previous commit. Nginx was still root, the namespace had no labels. The script was fixed to wait until each app reported the merge commit as its revision, not just "Synced".

Kubernetes won't change a binding's role

  monitoring        OutOfSync Progressing revision 92.18.0      after 381s

Argo CD couldn't change Grafana's binding:
  one or more synchronization tasks completed unsuccessfully, reason: ClusterRoleBinding.rbac.authorization.k8s.io "monitoring-grafana-clusterrolebinding" is invalid: roleRef: Invalid value: {"APIGroup":"rbac.authorization.k8s.io","Kind":"ClusterRole","Name":"grafana-read-configmaps"}: cannot change roleRef. Retrying attempt #5 at 10:27PM.

$ kubectl delete clusterrolebinding monitoring-grafana-clusterrolebinding
clusterrolebinding.rbac.authorization.k8s.io "monitoring-grafana-clusterrolebinding" deleted
  monitoring        Synced Progressing after 566s

A binding's roleRef can't be changed after it's created, on purpose: otherwise anyone allowed to edit bindings could quietly point one at cluster-admin. The only way is to delete it and make it again. Argo CD retried five times, then the binding was deleted, and Argo CD recreated it from Git, pointing at the new role. (Argo CD can do this itself, with the Replace=true sync option on that resource.)

A log collector with no access at all

During the first, too-early check, Alloy's log said:

$ kubectl logs -n monitoring deploy/alloy -c alloy --since=5m | grep -ciE 'forbidden'
22

  alloy                list secrets                 no
  alloy                list configmaps              no
  alloy                get pods --subresource=log   no

Alloy couldn't read logs either. Two Argo CD apps were involved: alloy (which removed the chart's role) and monitoring-config (which adds the new one). The first synced before the second, so for a while Alloy had no role at all. In a real cluster, that's a gap in the logs, exactly when nobody's looking. The Spot the problem below is about this.

7. After

Once everything had applied:

$ kubectl get ns bookshop-staging --show-labels
bookshop-staging   Active   2d23h   kubernetes.io/metadata.name=bookshop-staging,pod-security.kubernetes.io/enforce=restricted,pod-security.kubernetes.io/warn=restricted

Inside nginx

$ id
uid=101(nginx) gid=101(nginx) groups=101(nginx)

$ grep -E "^(CapEff|NoNewPrivs|Seccomp):" /proc/1/status
CapEff:	0000000000000000
NoNewPrivs:	1
Seccomp:	2

$ touch /etc/nginx/written-by-an-attacker; touch /tmp/scratch && echo "/tmp is writable (an emptyDir)"
touch: cannot touch '/etc/nginx/written-by-an-attacker': Read-only file system
/tmp is writable (an emptyDir)

$ ls /var/run/secrets/kubernetes.io/serviceaccount/
ls: cannot access '/var/run/secrets/kubernetes.io/serviceaccount/': No such file or directory

• User 101, not root.
• CapEff: 0: no capabilities at all.
• NoNewPrivs: 1: it can never gain more.
• Seccomp: 2: a filter is active (2 means "filter"; 0 was "none").
• Read-only, except the three emptyDir folders.
• No token.

nginx's own log noticed two things, both harmless:

10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
...
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2

The image's start-up script tries to edit its own configuration, can't, and carries on. And nginx's configuration says to switch to the nginx user, which it already is.

The namespace refuses what doesn't meet the standard

$ kubectl run root-test -n bookshop-staging --image=nginx:1.29 --dry-run=server
Error from server (Forbidden): pods "root-test" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "root-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "root-test" must set securityContext.capabil...

A plain nginx pod, the way most people would start one, is now forbidden in staging. That's what keeps the next change from quietly undoing this one.

Who can reach the bookshop

  from default     http://bookshop.bookshop-staging.svc/    000
  from default     https://bookshop.bookshop-staging.svc/   000
  from default     http://bookshop.bookshop.svc/            200
  from monitoring  http://bookshop.bookshop-staging.svc/    200
  from monitoring  https://bookshop.bookshop-staging.svc/   200
  from monitoring  http://bookshop.bookshop.svc/            200

From default, staging times out (000, curl's exit code 28); from monitoring, it answers. The connection isn't refused, it's dropped, so the caller waits. And the policy is enforced: kind's network plugin, kindnet, supports NetworkPolicies. Not every network plugin does, and a policy nothing enforces looks exactly like one that works until you test it. Production, untouched, still answers everyone.

Who can read every Secret

  argocd:argocd-application-controller
  argocd:argocd-applicationset-controller
  cert-manager:cert-manager
  cert-manager:cert-manager-cainjector
  kube-system:generic-garbage-collector
  kube-system:namespace-controller
  kube-system:resourcequota-controller
  kube-system:sealed-secrets-controller
  monitoring:monitoring-kube-prometheus-operator
  monitoring:monitoring-kube-state-metrics

  alloy                list secrets                 no
  alloy                list configmaps              no
  alloy                get pods --subresource=log   yes
  monitoring-grafana   list secrets                 no
  monitoring-grafana   list configmaps              yes
  monitoring-grafana   get pods --subresource=log   no

Twelve down to ten, and each of the two can do exactly its job. The two left in monitoring are worth a look next: the Prometheus operator manages Alertmanager's configuration Secrets, so it needs some access; kube-state-metrics reports on Secrets (names and types, for dashboards) and can be told not to.

Does monitoring still work?

$ kubectl exec -n monitoring deploy/monitoring-grafana -c grafana-sc-dashboard -- ls /tmp/dashboards | wc -l
      26

$ kubectl logs -n monitoring deploy/monitoring-grafana -c grafana-sc-dashboard --since=3m | grep -ciE 'forbidden|error'
2

$ kubectl logs -n monitoring deploy/alloy -c alloy --since=2m | grep -ciE 'forbidden'
0

$ curl ... loki ... 'sum(count_over_time({namespace="bookshop-staging"}[5m]))'
62

up bookshop 10.244.1.90:9113 = 1
probe_success bookshop-banner http://bookshop.bookshop-staging.svc/ = 1
probe_success bookshop-up http://bookshop.bookshop-staging.svc/ = 1

• Grafana: 26 dashboards loaded from ConfigMaps. Two log lines matched "error", and they were worth reading rather than assuming:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.14/site-packages/healthz.py", line 179, in _create_health_http_server
    return ThreadingHTTPServer(("", health_port), HealthHandler)
  ...
OSError: [Errno 98] Address in use

Not a permission problem (that would say Forbidden). The dashboard sidecar starts a small health-check server, and so does the data-source sidecar next to it. Containers in a pod share one network, so the second to start finds the port taken. The data-source sidecar's log had none. A quirk of how the two sidecars are set up, not of this change, and nothing to do with loading dashboards.

• Alloy: no forbidden errors, and Loki has staging's logs.
• Prometheus: still scraping the bookshop's exporter, and both blackbox probes succeed, through the NetworkPolicy, from monitoring.

Taking access away is only finished when what depended on it is shown to still work.

Spot the problem

Work out what's wrong before you read the explanation.

The tidy-up that blinded the alerts. A team reviews their log collector's permissions, sees it can read every Secret, and opens one pull request: turn off the chart's role, add a smaller one in their own config repository. It's reviewed, merged, and Argo CD applies it. Everything is green within ten minutes. A week later, during an incident investigation, they find twenty minutes of missing logs from the morning of the change, across every namespace. What happened? The old access was removed before the new access existed. Two Argo CD apps were involved, and they synced in whatever order Argo CD reached them. The collector's chart app removed its role first, and the config app with the new role followed later. In between, the collector could read nothing, and said so only in its own log, which it couldn't ship. This lesson had exactly that gap: 22 forbidden errors from Alloy, with "get pods/log: no". Three fixes: expand, then contract: add the new role in one change, check it works, and only then remove the old one; when one app has to go first, make it explicit (one app, with sync waves, or the order of merges); and alert on the collector's own health, such as Alloy's errors or a sudden drop in log volume, so a gap pages someone instead of waiting to be found.

Where you are now

You can check any namespace against the Pod Security Standards, take a container down to only what it needs, narrow a chart's role to what your setup uses, and limit who can connect, and prove each one from inside and outside. Staging is done; production is one values file away, and the same pull request pattern. The next lesson asks the question this one raised: when something does go wrong, would the logs show it? Logging for security, starting with the Kubernetes audit log Lesson 1 found missing.

Check yourself

Key terms

• Least privilege: giving everything only the access its job needs.
• Pod Security Standards: Kubernetes's three built-in levels for pods: privileged, baseline and restricted.
• Pod Security Admission: the part of the API server that enforces them, by namespace label.
• securityContext: the settings that decide a pod's or container's user, capabilities and file system access.
• Capabilities: the separate pieces root's power is divided into in Linux.
• seccomp: a filter on which system calls a process may make.
• emptyDir: an empty folder that lives as long as its pod.
• RBAC: role-based access control: Roles and ClusterRoles say what's allowed, bindings say who gets them.
• NetworkPolicy: rules for which pods may connect to which.
• Expand and contract: add the new thing, move to it, then remove the old one.

Go deeper

• Kubernetes: Pod Security Standards (opens in a new tab) and Pod Security Admission (opens in a new tab).
• Kubernetes: Configure a security context (opens in a new tab) and RBAC good practices (opens in a new tab).
• The NetworkPolicy editor (opens in a new tab) from Isovalent, for drawing policies and seeing what they allow.
• Try: set hardened: true and networkPolicy.enabled: true in values-production.yaml, with managedNamespaceMetadata on the production Application. What breaks, if anything?
• Try: add a default-deny egress policy to staging. What does the bookshop stop being able to reach, and does it matter?