What security work is, and a first look

What security work is

Films make security look like hacking. Most of the job is quieter. A security engineer spends their days on questions like these:

• What do we have? You can't protect what you don't know about. An up-to-date inventory is the first control, not paperwork.
• Who and what can do what? Every person, program and pod should have exactly the access its job needs. That's least privilege, and most breaches are a story of something having more than it needed.
• Is it up to date? Most attacks use weaknesses that already have a fix. Patching is security work.
• Where are the secrets? Passwords, keys and tokens end up in Git history, logs, screenshots and chat. Keeping them out, and replacing them when they leak, is daily work.
• Would we notice? Logs, alerts and audit trails. An attack you can't see is one you can't stop.
• Could we recover? Backups that have actually been restored, and a plan for when something goes wrong.

None of that is new to you if you've done the earlier tracks. Sealed Secrets, the limited Grafana login, Alertmanager and the dead-man's switch were all security work, even when the lesson was about something else.

What you're protecting

Every security decision protects one or more of three things, the CIA triad:

• Confidentiality: only the right people can read it. A leaked database password breaks confidentiality.
• Integrity: nobody can change it without permission, and you'd know if they did. A tampered image, or a silenced alert, breaks integrity.
• Availability: it's there when it's needed. A pod with no memory limit that takes down its node breaks availability.

A useful habit: for every finding, ask which of the three it puts at risk. It tells you how much it matters.

Defense in depth

No single control is enough, because every control eventually fails: a password leaks, a rule has a gap, a patch is late. Defense in depth means layers, so one failure isn't a breach. For a web app on Kubernetes, the layers look like this:

1. The edge: what's reachable from the internet at all.
2. The network inside: what each pod can talk to.
3. The workload: what each container is allowed to do, and as which user.
4. Identity: what each service account and person can do through the Kubernetes API.
5. The data: secrets encrypted, backups encrypted, nothing sensitive in logs.
6. Detection: logs and audit trails, so a failure in any layer is noticed.

The first look below checks each layer on the practice cluster.

A first look

You've just been handed a cluster and asked "is this secure?" The first answer is always "let me look". That means a read-only inventory: change nothing, write everything down. The script for this lesson, security01.sh, runs eight checks with kubectl, jq and curl. Every command and its real output is below.

What's here

NAME                   STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE
gitops-control-plane   Ready    control-plane   29h   v1.35.0   172.18.0.2    <none>        Debian GNU/Linux 12 (bookwor
gitops-worker          Ready    <none>          29h   v1.35.0   172.18.0.3    <none>        Debian GNU/Linux 12 (bookwor

kubectl get pods -A --no-headers | awk '{n[$1]++} END {for (k in n) print n[k], k}' | sort -k2

• awk '{n[$1]++}' counts lines by their first column, the namespace, and prints each count at the end.

7 argocd
2 bookshop
1 bookshop-staging
1 guestbook
11 kube-system
1 local-path-storage
11 monitoring

34 pods and, from the pod specs, 42 containers. Two nodes, Kubernetes 1.35. That's the inventory.

How each container is set to run

A pod's spec can say, for each container, which user it runs as and what it's allowed to do. If it says nothing, the image decides, and many images default to root. The script turns every container's settings into one line with jq:

kubectl get pods -A -o json | jq -r '
  .items[] | . as $p
  | ($p.spec.securityContext // {}) as $ps
  | $p.spec.containers[] as $c
  | ($c.securityContext // {}) as $cs
  | [ $p.metadata.namespace, $c.name,
      (if ($cs.runAsNonRoot // $ps.runAsNonRoot) == true then "nonroot" else "may-be-root" end),
      (if $cs.privileged == true then "PRIVILEGED" else "-" end),
      (if $cs.allowPrivilegeEscalation == false then "no-escalation" else "escalation-allowed" end),
      (if $cs.readOnlyRootFilesystem == true then "ro-root" else "writable-root" end)
    ] | @tsv'

• $cs // $ps: a setting can be on the container or on the whole pod. // means "this, or if it's missing, that".
• runAsNonRoot: Kubernetes refuses to start the container if its user would be root.
• privileged: the container gets almost everything the node itself can do.
• allowPrivilegeEscalation: false: a process can't gain more rights than it started with, for example through a setuid program.
• readOnlyRootFilesystem: the container can't change its own files, so an attacker can't drop tools into it.

(The full script also extracts memory limits, the node's network and the node's folders.) Some of what it found, trimmed to the first seven columns:

argocd       argocd-server            argocd-server       nonroot      -  no-escalation       ro-root
bookshop     bookshop                 exporter            may-be-root  -  escalation-allowed  writable-root
bookshop     bookshop                 nginx               may-be-root  -  escalation-allowed  writable-root
guestbook    guestbook-ui             guestbook-ui        may-be-root  -  escalation-allowed  writable-root
monitoring   alert-receiver           receiver            may-be-root  -  escalation-allowed  writable-root
monitoring   alloy                    alloy               may-be-root  -  escalation-allowed  writable-root
monitoring   prometheus-monitoring-kube-prometheus  prometheus  nonroot  -  no-escalation  ro-root

And the totals, for every namespace, kube-system included:

  containers 36 | may run as root 15 | privileged 1 | escalation allowed 15 | writable root filesystem 14 | no memory limit 28 | host network 7 | host paths 7

Two pictures in one cluster. Everything from a well-kept Helm chart, Argo CD, Prometheus, Alertmanager, Grafana and Loki, is set carefully: not root, read-only, no escalation. Everything written for these tracks, the bookshop, the alert receiver, and the old guestbook, plus Alloy, sets nothing at all. The defaults are the problem: Kubernetes doesn't make you choose.

Two notes on the numbers. 36, not 42: the script removes identical lines, so production's two identical pods count once; the totals are unique containers, not running copies. The privileged container and the host network and host path ones are in kube-system and monitoring's node exporter: kube-proxy, kindnet and the node exporter need the node's network or folders to do their jobs. That's expected, and why those namespaces get extra care rather than being the first worry.

Who they really run as

A spec that says nothing means "whatever the image says". To know, ask the container. The script asked five:

kubectl exec -n bookshop deploy/bookshop -c nginx -- id

bookshop           deploy/bookshop                nginx      uid=0(root) gid=0(root) groups=0(root)
bookshop-staging   deploy/bookshop                nginx      uid=0(root) gid=0(root) groups=0(root)
monitoring         deploy/alert-receiver          receiver   uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)

Root, confirmed. nginx's main process too:

Name:	nginx
Uid:	0	0	0	0

(From /proc/1/status, because the nginx image has no ps.) For Grafana and Loki, id printed nothing: their images are distroless, with no shell and no everyday tools at all. That's a security feature in itself. An attacker who gets into a distroless container finds no shell, no curl, nothing to work with.

Why root matters inside a container: if an attacker gets code running in nginx, through a bug in a module, say, root inside the container is one weakness away from root on the node. Containers share the node's kernel. A non-root user, a read-only filesystem and no escalation each make that step harder.

What's exposed outside the cluster

$ kubectl get svc -A -o json | jq -r '.items[] | select(.spec.type != "ClusterIP") | ...'

$ kubectl get ingress,gateway -A 2>/dev/null | head -5

$ docker ps --filter name=gitops --format '{{.Names}}  {{.Ports}}'
gitops-worker
gitops-control-plane  127.0.0.1:58154->6443/tcp

Nothing. No Service of type NodePort or LoadBalancer, no Ingress or Gateway, and the only published port is the Kubernetes API, on 127.0.0.1: the laptop's own address, unreachable from anywhere else. Every lesson has reached Grafana and Prometheus through kubectl port-forward, which needs a login to the Kubernetes API. The edge layer is closed.

What can talk to what

Inside a cluster, by default, every pod can reach every other pod. A NetworkPolicy is a firewall rule for pods: once a pod is selected by one, only the traffic it allows gets through.

$ kubectl get networkpolicies -A
NAMESPACE   NAME                                              POD-SELECTOR                                              AGE
argocd      argocd-application-controller-network-policy      app.kubernetes.io/name=argocd-application-controller      29h
argocd      argocd-applicationset-controller-network-policy   app.kubernetes.io/name=argocd-applicationset-controller   29h
argocd      argocd-dex-server-network-policy                  app.kubernetes.io/name=argocd-dex-server                  29h
argocd      argocd-notifications-controller-network-policy    app.kubernetes.io/name=argocd-notifications-controller    29h
argocd      argocd-redis-network-policy                       app.kubernetes.io/name=argocd-redis                       29h
argocd      argocd-repo-server-network-policy                 app.kubernetes.io/name=argocd-repo-server                 29h
argocd      argocd-server-network-policy                      app.kubernetes.io/name=argocd-server                      29h

Argo CD ships its own, one per component. Nothing else has any. And Pod Security admission, Kubernetes's built-in check that refuses risky pods, is set on no namespace: the ENFORCE column was empty for all ten.

So the test: a throwaway curl pod in bookshop-staging, the least important namespace, as if an attacker had got in through staging's web server.

kubectl run sec-probe -n bookshop-staging --image=curlimages/curl:8.11.1 --restart=Never --command -- sleep 600
kubectl exec -n bookshop-staging sec-probe -- curl -s -o /dev/null -m 5 -w '%{http_code}' http://bookshop.bookshop.svc/

• kubectl run ... -- sleep 600 starts a pod that just waits, so commands can be run in it.
• -w '%{http_code}' prints only the HTTP status: 200 means it answered.

  the production shop (bookshop namespace)                   200
  Grafana                                                    200
  Prometheus's query API                                     200
  Alertmanager's API                                         200
  Loki's API (every pod's logs)                              200
  Argo CD's web interface                                    307
  the Sealed Secrets controller                              200

Everything answered. (307 is Argo CD redirecting to HTTPS; its login still protects it. The Sealed Secrets controller serves its public certificate, which is meant to be public.) Reaching a service isn't the same as being able to use it, so the next question: what can it do with no login at all?

{"targets_prometheus_scrapes":"26"}
{"namespaces_whose_logs_it_can_read":["argocd","bookshop","bookshop-staging","kube-system","local-path-storage","monitoring"]}

Every metric, and every namespace's logs. Prometheus's and Loki's APIs have no login, because they're designed to sit behind something that has one. And Alertmanager:

Could it silence every alert? A one-minute silence that matches nothing real:
  silence created from the test pod: 9fbe880f-c418-44c6-9acd-687fe5df3f9a
  removed: HTTP 200

A pod in staging can silence alerts. This one matched an alert that never fires, SecurityLesson1Demo, for one minute, and was deleted at once. A real attacker would silence everything, for a day, before doing anything else, and the monitoring from the last track would stay quiet. That's integrity, not just confidentiality.

The pod's own credentials, last. Kubernetes gives every pod a service account token by default:

$ ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt
namespace
token

$ curl ... -H "Authorization: Bearer $(cat .../token)" https://kubernetes.default.svc/api/v1/namespaces/bookshop-staging/secrets
403  (listing secrets in its own namespace)

403, forbidden. The token exists, but the default service account can do almost nothing with it. That layer held. Still, a token a pod doesn't need is better not mounted at all (automountServiceAccountToken: false), so there's nothing to steal.

Who can do everything

cluster-admin: Group/-/system:masters
kubeadm:cluster-admins: Group/-/kubeadm:cluster-admins

Only the two groups kind's own admin login belongs to have cluster-admin, Kubernetes's "can do anything" role. That's your kubeconfig, which is right. But which service accounts can read every Secret in the cluster?

kubectl auth can-i list secrets -A --as=system:serviceaccount:monitoring:alloy

• kubectl auth can-i ... --as= asks the API server "if I were this account, could I do this?", without needing that account's token.

  argocd:argocd-application-controller          read secrets in every namespace: yes
  kube-system:sealed-secrets-controller         read secrets in every namespace: yes
  monitoring:alloy                              read secrets in every namespace: yes
  monitoring:monitoring-grafana                 read secrets in every namespace: yes
  bookshop:default                              read secrets in every namespace: no

Four, and two of them shouldn't. Argo CD applies Secrets everywhere, and the Sealed Secrets controller makes them: both need it. But Alloy, the log collector, and Grafana, the dashboards, can also read every Secret: the Grafana admin password, the Slack webhook, the heartbeat address, all of it. Their charts grant it so optional features work out of the box: Alloy can read Secrets for some of its components, and Grafana's sidecar can watch Secrets for dashboards and data sources. Neither uses it here. If either is ever compromised, every Secret goes with it.

How Secrets are kept

3 argocd
1 bookshop
2 kube-system
14 monitoring

$ kubectl get pod -n kube-system -l component=kube-apiserver -o json | jq -r '.items[0].spec.containers[0].command[]' | grep -c -e encryption-provider-config -e audit-log-path
0

• The second command reads the API server's startup flags, looking for the two that turn on encryption of Secrets at rest and an audit log. 0 means neither is set.

$ kubectl get secret grafana-admin -n monitoring -o jsonpath='{.data.admin-user}' | base64 -d
admin  <- a Secret is base64, not encryption

A Kubernetes Secret is only encoded, not encrypted: anyone who can read it, or can read the database behind the API server, etcd, can decode it. Sealed Secrets protects them in Git; once they're in the cluster, they're plain. Encryption at rest protects them in etcd. And without an audit log, there's no record of who read or changed what through the API: if a Secret were read by the wrong account, nothing would say so.

What's running, by version

images: tag only 28, pinned by digest 1, latest or no tag

(The blank after "latest or no tag" is zero: the awk only prints a number once it has counted one.) A sample of the list:

docker.io/grafana/alloy:v1.21.3
docker.io/grafana/grafana:13.3.0-distroless
gcr.io/google-samples/gb-frontend:v5
nginx:1.29
python:3.13-alpine
quay.io/prometheus-operator/prometheus-config-reloader:v0.94.2@sha256:a1897c67478bc23434534e91ad3855280771196b1daf17144db9dbcb655bef71
quay.io/prometheus/prometheus:v3.15.1-distroless

Nothing on latest, which is good, and every version pinned in Git. But a tag can be moved: nginx:1.29 today may not be the same image as nginx:1.29 next month. A digest, @sha256:..., names exactly one image, forever. And nothing here checks the images for known vulnerabilities. gb-frontend:v5, the guestbook from the Argo CD track's first lesson, is a sample app nobody has looked at since.

The findings, and the plan

Every first look ends the same way: a list, each finding rated, each with an owner. Here, the owner is the rest of this track.

And what's already right, worth writing down too: nothing reachable from outside, Argo CD hardened and fenced in by its own policies, the Prometheus stack running non-root and read-only, distroless images where the charts offer them, no latest tags, a default service account that can't read Secrets, and Secrets sealed before they reach Git. A first look that finds only problems hasn't looked properly.

Spot the problem

Work out what's wrong before you read the explanation.

"It's internal, so it doesn't need a login." A team runs Prometheus, Loki and Alertmanager in their production cluster with no authentication, because none of them is reachable from the internet. Their web shop has a bug that lets an attacker run commands inside its pod. The attacker never touches the internet-facing parts again. What can they do from there? Everything this lesson's test pod did. Read every metric (which reveal the architecture, the versions and the traffic), read every log (which often hold user emails, tokens and error messages with passwords in them), and silence every alert before going further. "Not reachable from outside" protects only the first layer; once one pod falls, everything the network lets it reach falls next. The habit: assume a pod will be compromised one day, and ask what it could reach from there. NetworkPolicies limit the reach (Lesson 9); logins on internal APIs and audit logs limit what can be done and show it happening (Lesson 10).

Where you are now

You know what security work is, the three things it protects, and how layers keep one failure from becoming a breach. And you have a written first look at the practice cluster: closed from outside, open inside, with ten findings and a lesson for each. Before fixing any of them, the next lesson asks a better question than "what's wrong?": what would an attacker actually want, and how would they get it? That's threat modeling, and it decides which findings to fix first.

Check yourself

Key terms

• CIA triad: confidentiality, integrity and availability, the three things security protects.
• Defense in depth: layers of controls, so one failing isn't a breach.
• Least privilege: every person, program and pod has only the access its job needs.
• securityContext: the part of a pod spec that sets the user, privileges and filesystem rules for its containers.
• Distroless image: a container image with the program and nothing else: no shell, no tools.
• NetworkPolicy: a firewall rule for pods, saying which traffic may reach them.
• Pod Security admission: Kubernetes's built-in check that refuses pods breaking a chosen security level.
• Service account token: the credential Kubernetes mounts into pods for talking to its API.
• Encryption at rest: stored data, here Secrets in etcd, kept encrypted on disk.
• Audit log: the API server's record of who did what, and when.

Go deeper

• The Kubernetes documentation on security concepts (opens in a new tab), Pod Security Standards (opens in a new tab) and NetworkPolicies (opens in a new tab).

• The OWASP Kubernetes Top 10 (opens in a new tab): the most common Kubernetes security risks, several of which this first look found.

• Try: list everything Alloy may do, then decide which of it a log collector reading pod logs through the API actually needs:

  BASHCopy

  kubectl auth can-i --list --as=system:serviceaccount:monitoring:alloy
  

• Try: from the test pod, ask Prometheus for something sensitive-looking, such as kube_secret_info. What does it tell an attacker, even without the Secrets' values?