Logging for security

By the end you'll know

What security logging is for, and what makes an event worth keeping
What the Kubernetes audit log records, and how an audit policy decides what goes in it
How to read an audit event: who, what, when, from where, and why it was allowed
How to find an incident in Loki with LogQL, and turn the query into an alert
Why a log's blind spots matter as much as what it records

Before you start

You need: the cluster, Loki, Alloy and Alertmanager from the Monitoring track, and Lesson 9 done.
This lesson changes the API server (the audit log stays on, the same kind of change as Lesson 5) and adds alert rules through a pull request.
Maps to Security+: domain 4 (security monitoring, log data, alerting, investigation).

The short version

Before: nothing recorded requests to the API server. A Secret could be read by anyone allowed to, and nothing would say who.
The audit log: one policy file, three flags, 46 seconds without an API server. About 180 events a minute with nothing happening.
The incident: a stolen token reads a payment key, then tries kube-system and the practice CA's key. The audit log names the account, the pod it came from, the address, the tool (curl) and the role that allowed it. The key's value appears nowhere.
Loki already had it, because the audit log goes to the API server's own output, which Alloy ships.
Two alert rules in LogQL: five alerts in Alertmanager, and Slack, 41 seconds after the incident.
The gap: a refused attempt to list the cluster's role bindings left no trace at all.

What security logging is for

Monitoring (the last track) answers "is it working?". Security logging answers different questions, usually afterwards and under pressure:

Who did this? An account, and behind it a person or a program.
What exactly did they do, and what were they refused?
When, and from where?
What else did they touch?

For that, a log needs to be complete for the events that matter, attributed (an account, not "someone"), kept somewhere the actor can't change it, and searched and alerted on, because nobody reads logs until something has already gone wrong. Lesson 4 did this for one server with auditd. This lesson does it for the cluster's API, where every change and every Secret read passes through.

1. Before

OUTPUT

$ kubectl -n kube-system get pod kube-apiserver-gitops-control-plane -o yaml | grep -cE 'audit-(policy-file|log-path|webhook)'
0

No audit log, no audit webhook. Lesson 1 found this, and every lesson since has touched Secrets with nothing to show who.

2. An audit policy

The API server can record every request, at four levels:

None: not recorded.
Metadata: who, what, when, from where, and the answer. Not the body of the request or the response.
Request: that, plus what was sent.
RequestResponse: that, plus what came back.

A policy says which level each request gets. The first rule that matches wins:

YAML

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps", "serviceaccounts/token"]
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward", "pods/proxy", "services/proxy"]
  - level: None
    resources:
      - group: "coordination.k8s.io"
        resources: ["leases"]
      - group: ""
        resources: ["events"]
  - level: Metadata
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
  - level: None

Secrets, ConfigMaps and service account tokens at Metadata, never higher. Request or RequestResponse would copy every Secret's value into the log, and the log would become the easiest place to steal them from.
Shells and connections into pods (kubectl exec, port-forward): someone getting inside.
Leader-election heartbeats and events: not recorded. They arrive every few seconds and say nothing about security.
Every change to anything else at Metadata.
Everything else, mostly reads: not recorded. That keeps the volume down. It's also the gap this lesson ends on.
omitStages: ["RequestReceived"]: record each request once, when it's answered, not also when it arrives.

3. Turning it on

Three flags on the API server, and the policy mounted into it, the same way Lesson 5 added encryption:

OUTPUT

>     - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
>     - --audit-log-path=-
>     - --audit-log-format=json
>     - mountPath: /etc/kubernetes/audit
>       name: audit-policy
>       readOnly: true

Waiting for the kubelet to restart the API server...
API server back after about 46 seconds

--audit-log-path=- writes the audit log to the API server's own output instead of a file. That output is a pod's log, so Alloy, which already ships every pod's log to Loki, picks it up with nothing new installed. (The usual alternatives are a file on the control plane node, collected by an agent there, or --audit-webhook-config-file to send events straight to a collector.)
json: one event per line, easy to search.

How much it records

OUTPUT

$ audit 1m | wc -l
     183

  76 watch	configmaps
  57 watch	secrets
  16 patch	pods
  10 create	serviceaccounts
   6 create	subjectaccessreviews
   3 patch	applications
   3 delete	pods
   2 update	endpointslices

  79 system:node:gitops-worker
  26 system:node:gitops-control-plane
  14 system:serviceaccount:argocd:argocd-application-controller
  12 system:kube-controller-manager
  11 system:serviceaccount:monitoring:monitoring-kube-prometheus-operator
   8 system:serviceaccount:argocd:argocd-applicationset-controller

About 180 events a minute, with nothing happening. Most are the two nodes' kubelets watching the Secrets and ConfigMaps their pods mount. On a real cluster with hundreds of nodes and busy controllers, the same policy produces millions of events a day, which is why policies are tuned, and why the log goes somewhere built to store it.

4. An incident: a stolen token

A reporting job's service account may read Secrets in its own namespace, default, where a payment provider's key is kept. Someone gets the token (from a leaked pod, a copied file, a debug session) and tries it from inside a pod with curl:

OUTPUT

/api/v1/namespaces/default/secrets                             200
/api/v1/namespaces/default/secrets/payments-api-key            200
/api/v1/namespaces/kube-system/secrets                         403
/api/v1/namespaces/cert-manager/secrets/practice-root-ca       403
/apis/rbac.authorization.k8s.io/v1/clusterrolebindings         403

Two successes, then three refusals: it got what the role allowed, and tried for more. (RBAC did its job; the payment key is still gone.) Separately, an administrator opens a shell in the bookshop with kubectl exec.

5. What the audit log recorded

OUTPUT

{"time":"22:44:10","user":"system:serviceaccount:default:reporting","verb":"list","resource":"default/secrets/","code":200,"from":"172.18.0.3","agent":"curl/8.14.1"}
{"time":"22:44:10","user":"system:serviceaccount:default:reporting","verb":"get","resource":"default/secrets/payments-api-key","code":200,"from":"172.18.0.3","agent":"curl/8.14.1"}
{"time":"22:44:10","user":"system:serviceaccount:default:reporting","verb":"list","resource":"kube-system/secrets/","code":403,"from":"172.18.0.3","agent":"curl/8.14.1"}
{"time":"22:44:10","user":"system:serviceaccount:default:reporting","verb":"get","resource":"cert-manager/secrets/practice-root-ca","code":403,"from":"172.18.0.3","agent":"curl/8.14.1"}
{"time":"22:44:10","user":"kubernetes-admin","verb":"get","resource":"bookshop-staging/pods/bookshop-689f6b789f-d8mmc/exec","code":101,"from":"172.18.0.1","agent":"kubectl/v1.33.3 (darwin/arm64)"}

(A summary of each event, made with jq; the script's own kubectl exec calls into the reporting pod are recorded too, and left out here.) Each line is a request: who, what, the answer (101 is a connection switching to a shell), the address and the program. The Secret read, in full:

OUTPUT

{
  "level": "Metadata",
  "auditID": "54740778-d54e-41bd-a3a0-540441ce8941",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/default/secrets/payments-api-key",
  "verb": "get",
  "user": {
    "username": "system:serviceaccount:default:reporting",
    "groups": ["system:serviceaccounts", "system:serviceaccounts:default", "system:authenticated"],
    "extra": {
      "authentication.kubernetes.io/credential-id": ["JTI=c0e08bae-d168-4b62-8922-b05b452c3360"],
      "authentication.kubernetes.io/node-name": ["gitops-worker"],
      "authentication.kubernetes.io/pod-name": ["reporting-job"],
      ...
    }
  },
  "sourceIPs": ["172.18.0.3"],
  "userAgent": "curl/8.14.1",
  "objectRef": { "resource": "secrets", "namespace": "default", "name": "payments-api-key", "apiVersion": "v1" },
  "responseStatus": { "code": 200 },
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by RoleBinding \"reporting-read-secrets/default\" of Role \"read-secrets\" to ServiceAccount \"reporting/default\""
  }
}

What an investigator gets from one line:

Which token: credential-id is the token's own ID, so you can tell this token from others for the same account.
Which pod it was issued to: pod-name: reporting-job, on node gitops-worker. Modern service account tokens are bound to a pod, and the audit log says which.
Why it was allowed: the exact RoleBinding and Role. That's where the fix goes.
What it wasn't: a browser or kubectl. A reporting job doesn't normally use curl.

And what it doesn't contain:

OUTPUT

$ kubectl logs -n kube-system kube-apiserver-gitops-control-plane --since=10m | grep -c 'sk_live_lesson10'
0
$ kubectl logs ... | grep -c "$(printf %s sk_live_lesson10_do_not_log_me | base64)"
0

The key's value is nowhere, as text or base64. Metadata did what it promised.

6. The same events, in Loki

LogQL, Loki's query language, can parse each line as JSON and filter on its fields. Nested fields become names joined by underscores: user.username becomes user_username.

BASH

{namespace="kube-system", container="kube-apiserver"} |= "audit.k8s.io" | json | user_username="system:serviceaccount:default:reporting"

{namespace=..., container=...}: the API server's log.
|= "audit.k8s.io": only audit events, not its other messages. A text filter first is much faster than parsing every line.
| json: parse the event.
| user_username="...": keep this account's requests.

OUTPUT

22:44:10	get	/api/v1/namespaces/cert-manager/secrets/practice-root-ca	403
22:44:10	get	/api/v1/namespaces/default/secrets/payments-api-key	200
22:44:10	list	/api/v1/namespaces/default/secrets	200
22:44:10	list	/api/v1/namespaces/kube-system/secrets	403

And who asks for Secrets at all, counted over ten minutes:

BASH

sum by (user_username, verb, responseStatus_code) (count_over_time({...} |= "audit.k8s.io" | json | objectRef_resource="secrets" [10m]))

OUTPUT

system:node:gitops-worker                                             watch   200  24
system:serviceaccount:argocd:argocd-application-controller            watch   200  10
system:serviceaccount:argocd:argocd-applicationset-controller         watch   200  10
system:serviceaccount:argocd:argocd-server                            watch   200  4
system:serviceaccount:cert-manager:cert-manager                       watch   200  4
...
system:serviceaccount:kube-system:sealed-secrets-controller           watch   200  3
system:serviceaccount:monitoring:monitoring-kube-prometheus-operator  watch   200  3
system:node:gitops-worker                                             watch   403  2
...
system:serviceaccount:default:reporting                               get     200  1
system:serviceaccount:default:reporting                               get     403  1
system:serviceaccount:default:reporting                               list    200  1
system:serviceaccount:default:reporting                               list    403  1

That's a baseline: the accounts that read Secrets for a living. Everything on it is a kubelet, the control plane, Argo CD, cert-manager, Sealed Secrets or the two monitoring components Lesson 9 left with access. The reporting account isn't, and neither is a person. The alert is built from exactly this list.

7. Alerts from the audit log

Loki has a ruler: it runs LogQL queries on a schedule and, like Prometheus, sends any that return results to Alertmanager, which already posts to Slack. Two rules, in a ConfigMap in apps/monitoring:

YAML

- alert: SecretReadByUnexpectedAccount
  expr: |
    sum by (user_username, objectRef_namespace, objectRef_name, responseStatus_code) (
      count_over_time({namespace="kube-system", container="kube-apiserver"}
        |= "audit.k8s.io" |= `"resource":"secrets"`
        | json
        | verb =~ "get|list"
        | user_username !~ "system:(node|kube-controller-manager|kube-scheduler|apiserver).*|system:serviceaccount:(argocd|cert-manager|kube-system):.*|system:serviceaccount:monitoring:monitoring-kube-(prometheus-operator|state-metrics)"
      [5m])
    ) > 0
- alert: ServiceAccountForbidden
  expr: |
    sum by (user_username, verb, objectRef_resource, objectRef_namespace) (
      count_over_time({namespace="kube-system", container="kube-apiserver"}
        |= "audit.k8s.io" |= `"code":403`
        | json
        | user_username =~ "system:serviceaccount:.*"
      [5m])
    ) > 0

The first: a get or list of Secrets by anyone not on the baseline. watch is left out, since it's how controllers keep up to date. People (kubernetes-admin) are deliberately not on the list: a person reading a Secret should be rare enough to notice.
The second: any service account refused anything. Programs don't usually ask for things they aren't allowed; a 403 from a service account usually means a misconfiguration or someone trying a token out. Both are worth knowing.
sum by (...): one alert per account and target, so the message says who and where.

Getting the rules into Loki

The Loki chart has an option for this, ruler.directories, and it's a trap: in the single-pod (Monolithic) mode this cluster uses, it mounts a ConfigMap the chart only creates in distributed mode, which would leave Loki unable to start. Reading the chart's templates found that before it broke anything. So the pull request turns the ruler on and mounts the rules itself:

YAML

          rulerConfig:
            alertmanager_url: http://monitoring-kube-prometheus-alertmanager.monitoring.svc:9093
            rule_path: /var/loki/rules-temp
            storage:
              type: local
              local:
                directory: /etc/loki/rules
        singleBinary:
          extraVolumes:
            - name: audit-rules
              configMap:
                name: loki-audit-rules
                optional: true
          extraVolumeMounts:
            - name: audit-rules
              mountPath: /etc/loki/rules/fake

/etc/loki/rules/fake: the ruler reads one folder per tenant, and with logins off, Loki's tenant is called fake.
optional: true: if Loki restarts before the ConfigMap exists, it starts without rules instead of waiting.

Before committing, the script rendered the real chart with these values:

OUTPUT

95:    ruler:
96:      alertmanager_url: http://monitoring-kube-prometheus-alertmanager.monitoring.svc:9093
97:      rule_path: /var/loki/rules-temp
100:          directory: /etc/loki/rules
101:        type: local

360:            name: loki-audit-rules
362:          name: audit-rules
425:            - mountPath: /etc/loki/rules/fake
426:              name: audit-rules

  name: loki
  name: loki-runtime

The ruler configured, the rules mounted, and the chart making only its own two ConfigMaps. It went in as pull request #77.

8. The incident again, and the alerts

After the merge, Argo CD applied it (Loki restarted in under a minute), and the stolen token was tried again:

OUTPUT

/api/v1/namespaces/default/secrets/payments-api-key            200
/api/v1/namespaces/kube-system/secrets                         403
/api/v1/namespaces/cert-manager/secrets/practice-root-ca       403

41 seconds after the incident, audit alerts in Alertmanager: 5

SecretReadByUnexpectedAccount
  system:serviceaccount:default:reporting read Secrets in default (answer 200)
  receivers: bookshop-team
SecretReadByUnexpectedAccount
  system:serviceaccount:default:reporting read Secrets in kube-system (answer 403)
  receivers: bookshop-team
SecretReadByUnexpectedAccount
  system:serviceaccount:default:reporting read Secrets in cert-manager (answer 403)
  receivers: bookshop-team
ServiceAccountForbidden
  system:serviceaccount:default:reporting was refused list secrets in kube-system
  receivers: bookshop-team
ServiceAccountForbidden
  system:serviceaccount:default:reporting was refused get secrets in cert-manager
  receivers: bookshop-team

Five alerts, 41 seconds after the incident, to the bookshop team's Slack channel. Each says which account and where. One thing to fix: the first rule's message says "read Secrets ... (answer 403)" for requests that were refused. "Asked for" would be accurate. Wording in an alert matters: whoever reads it at 3 a.m. will take it literally.

Two checks in the script didn't work, and the alerts made them unnecessary: Loki's image has no ls (no shell tools at all, like Grafana's in Lesson 9), and the query of Loki's rules API printed nothing. Rules that fire are rules that loaded.

In real systems

Managed clusters have an audit log already, sent to the cloud's own logging: CloudWatch on EKS (it has to be turned on), Cloud Logging on GKE, Azure Monitor on AKS. You choose what to keep and alert on.
Logs go to a SIEM (security information and event management system), such as Splunk, Elastic Security, Microsoft Sentinel or Google SecOps, alongside cloud, identity and network logs. An incident usually crosses all of them.
Keep the log out of the actor's reach. A separate account or project, write-once storage, and retention long enough for investigations, often a year. Attackers delete logs.
Start from a published policy. The Kubernetes documentation's example, and GKE's own, are good starting points; tune from what your volume shows.
Detection rules are code. Falco watches what containers do at runtime; Sigma rules describe detections in a shared format. Keep rules in Git, review them, and test them by replaying incidents, as this lesson did.
Every alert needs an owner and a next step. Here both go to the bookshop team; a real security alert goes to whoever can act on it, with a runbook.

Spot the problem

Work out what's wrong before you read the explanation.

The probe nobody saw. After an incident, a team searches their audit log for everything a stolen service account token did. They find two successful Secret reads and two refused ones, and conclude the attacker went straight for Secrets and stopped. Weeks later, a second token is used to change a role binding, and nobody can say how the attacker knew which binding to change. What did the log miss? The reconnaissance. In this lesson, the stolen token also asked to list every ClusterRoleBinding in the cluster, and was refused:

OUTPUT

/apis/rbac.authorization.k8s.io/v1/clusterrolebindings         403

It's in the token's own output, and nowhere in the audit log. The policy recorded Secret requests and changes, and dropped reads of everything else to keep the volume down. An audit policy matches on who, what and which resource, never on the answer, so "record refusals" can't be written as a rule. Every probe of something the policy doesn't cover is invisible, allowed or refused, and probing is what attackers do first. Three fixes: record reads of the sensitive kinds too (RBAC objects, service accounts, tokens, nodes), at Metadata; record every request from service accounts outside the system namespaces, since programs make few, predictable requests; and test the policy against an attack, as this lesson did, then check that every step of it shows up.

Keep this

To	Use
Turn on the audit log	`--audit-policy-file`, `--audit-log-path` (`-` for the API server's output), `--audit-log-format=json`
Record Secrets without their values	`level: Metadata`, never `Request` or `RequestResponse`
Who an event was	`.user.username`, and `.user.extra` for the token's pod
Why it was allowed	`.annotations["authorization.k8s.io/reason"]`
Audit events in Loki	`{container="kube-apiserver"} \|= "audit.k8s.io" \| json`
Filter on a nested field	`\| user_username="..."`, `\| objectRef_resource="secrets"`
Count by field	`sum by (FIELD) (count_over_time(... [10m]))`
Alert on logs	Loki's ruler, with `rulerConfig.alertmanager_url` and rules under `/etc/loki/rules/TENANT`
See what's firing	`curl ALERTMANAGER:9093/api/v2/alerts`

Where you are now

The cluster records who asks for Secrets and who gets into pods, keeps it in Loki next to everything else, and tells Slack within a minute when the wrong account tries. You know how to read an audit event down to the pod a token came from, and how to find what a policy can't see. The last lesson puts all of it to use: an incident, from the first alert to the last step, and what to do in what order when it's real.

Check yourself

Why are Secrets recorded at Metadata, not RequestResponse?
How did the audit event show which pod the stolen token came from?
Why did Loki already have the audit events, with nothing new installed?
Why does the first alert rule leave out watch requests?
The token's refused request for clusterrolebindings wasn't in the audit log. Why?
Why did the chart's ruler.directories option get avoided?

Key terms

Audit log: a record of every request to the Kubernetes API server, by policy.
Audit policy: the rules deciding what's recorded, and at which level.
Audit level: None, Metadata, Request or RequestResponse: how much of each request is kept.
Bound service account token: a token issued for one pod, carrying that pod's name, and expiring.
LogQL: Loki's query language: a stream selector, then filters and parsers.
Ruler: the part of Loki (or Prometheus) that runs queries on a schedule and sends alerts.
Baseline: what normal looks like, so the unusual stands out.
Reconnaissance: an attacker finding out what's there and what they can reach, before acting.
SIEM (security information and event management): a system that collects security logs from many sources and alerts on them.

Go deeper

Kubernetes: Auditing (opens in a new tab), including an example policy.
Grafana: LogQL (opens in a new tab) and alerting and recording rules in Loki (opens in a new tab).
Falco (opens in a new tab), for detecting what containers do at runtime, and its rules for Kubernetes audit events.
Sigma (opens in a new tab), a shared format for detection rules, with rules for Kubernetes.
Try: add rbac.authorization.k8s.io objects and serviceaccounts to the policy's first rule, restart the API server, and replay the incident. Does the probe show up now?
Try: fix the alert's wording so refused requests say "asked for", and add a third rule for kubectl exec into the bookshop.