Incident response basics

Incident response, in phases

An incident is anything that threatens the confidentiality, integrity or availability of a system: a breach, a leak, malware, a stolen credential, or an outage someone caused. Incident response is the practised way of dealing with one, so the decisions under pressure have mostly been made in advance.

The phases, as Security+ teaches them (from NIST's incident handling guide, SP 800-61):

1. Preparation: the logging, alerts, access controls, runbooks and people, in place before anything happens.
2. Detection: something notices.
3. Analysis (triage): what is it, how bad, is it still happening?
4. Containment: stop it getting worse.
5. Eradication: remove what the attacker left.
6. Recovery: back to normal, safely.
7. Lessons learned: write it up and fix the causes.

NIST's current version (Revision 3, 2025) folds these into its Cybersecurity Framework's functions: Govern, Identify, Protect, Detect, Respond and Recover. The work is the same. In practice the middle phases overlap and loop: you contain, find more, analyze again.

This whole track has been preparation. Lesson 4's baselines, Lesson 5's encrypted Secrets, Lesson 9's least privilege and Lesson 10's audit log and alerts are what this lesson uses.

1. How things were

The setup, done before the incident:

serviceaccount/reporting
role.rbac.authorization.k8s.io/read-secrets
rolebinding.rbac.authorization.k8s.io/reporting-read-secrets
secret/payments-api-key

A reporting job's service account that may read Secrets in default, where a payment provider's key is kept. And a contractor, Jordan, given a client certificate signed by the cluster's certificate authority, through Kubernetes's own certificate signing API, valid for two hours, with the built-in edit role in default:

NAME     SIGNER                                EXPIRES(s)   CONDITION
jordan   kubernetes.io/kube-apiserver-client   7200         Approved

$ kubectl --kubeconfig jordan.kubeconfig auth whoami
Username                                            jordan@contractor.example
Groups                                              [contractors system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [X509SHA256=b428e298fb50b3efbcd3924b649ac90591da72549cc387364d41d390b544b67e]

NAME                     ROLE               USERS
contractor-jordan-edit   ClusterRole/edit   jordan@contractor.example

All ordinary. Then the script did something with Jordan's credentials, at about 23:04:49 UTC, with its output hidden.

2. Detection

All the person on call had:

23:05:20  ServiceAccountForbidden
          system:serviceaccount:default:reporting was refused list secrets in kube-system
23:05:20  SecretReadByUnexpectedAccount
          system:serviceaccount:default:reporting read Secrets in default (answer 200)
23:05:20  SecretReadByUnexpectedAccount
          kubernetes-admin read Secrets in default (answer 200)
23:05:20  SecretReadByUnexpectedAccount
          system:serviceaccount:default:reporting read Secrets in kube-system (answer 403)
23:05:20  SecretReadByUnexpectedAccount
          jordan@contractor.example read Secrets in default (answer 200)
23:05:20  SecretReadByUnexpectedAccount
          jordan@contractor.example read Secrets in default (answer 200)

Six alerts, 31 seconds after the attack. Read them the way the person on call would:

• A contractor read Secrets. Possible, but unusual.
• A service account read Secrets, then tried kube-system's and was refused. A reporting job has no reason to look in kube-system. That's someone trying a token out.
• kubernetes-admin read Secrets. That was the setup script listing what it had made. Responders' and administrators' own actions trip alerts too; knowing which alerts are yours is part of triage.

What to do first: not fix anything. Find out what's going on, and whether it's still happening.

3. Triage

Every request the audit log has from Jordan, from Loki (Lesson 10):

23:04:49  jordan@contractor.example  create  -/selfsubjectreviews/             201  kubectl/v1.33.3 (darwi
23:04:49  jordan@contractor.example  list    default/secrets/                  200  kubectl/v1.33.3 (darwi
23:04:49  jordan@contractor.example  get     default/secrets/payments-api-key  200  kubectl/v1.33.3 (darwi
23:04:49  jordan@contractor.example  create  default/deployments/collector     201  kubectl/v1.33.3 (darwi
23:04:49  jordan@contractor.example  create  default/services/collector        201  kubectl/v1.33.3 (darwi
23:04:49  jordan@contractor.example  create  default/cronjobs/metrics-sync     201  kubectl/v1.33.3 (darwi

In one second: listed the Secrets, read the payment key, and created three things. (The first line, selfsubjectreviews, is kubectl auth whoami from the setup's own check, run with the same credentials.) A Deployment and Service called collector, and a CronJob (a job Kubernetes runs on a schedule) called metrics-sync. Names chosen to look routine.

And the reporting account:

22:44:10  sa:default:reporting  get   default/secrets/payments-api-key       200  curl/8.14.1
22:44:10  sa:default:reporting  list  kube-system/secrets/                   403  curl/8.14.1
22:44:10  sa:default:reporting  get   cert-manager/secrets/practice-root-ca  403  curl/8.14.1
22:53:47  sa:default:reporting  get   default/secrets/payments-api-key       200  curl/8.14.1
22:53:47  sa:default:reporting  list  kube-system/secrets/                   403  curl/8.14.1
22:53:47  sa:default:reporting  get   cert-manager/secrets/practice-root-ca  403  curl/8.14.1
23:05:00  sa:default:reporting  get   default/secrets/payments-api-key       200  curl/8.11.1
23:05:00  sa:default:reporting  list  kube-system/secrets/                   403  curl/8.11.1

       9
   2 metrics-sync-29901545-7kxbw
   7 reporting-job

A trap. The same account, the same Secret, the same refused requests, starting 20 minutes before Jordan did anything. It would be easy to decide the attack began at 22:44. It didn't: those were Lesson 10's tests, which used an account with the same name. Two things tell them apart: a different version of curl, and, decisively, the pod each token was issued to. Seven requests came from reporting-job, an old test pod; the two from this incident came from metrics-sync-29901545-7kxbw, a pod the CronJob made. Accounts get reused; pod names and token IDs don't.

So: Jordan's credentials made a job that runs as the reporting account every minute, reads the payment key and tries for more. Is it still happening? Yes, every minute.

4. Evidence first

Before changing anything, save what's there, because containment and eradication destroy it:

kubectl get cronjob,job,deployment,service,pod -n default -o yaml > objects-before.yaml
kubectl get rolebinding,role,serviceaccount -n default -o yaml > rbac-before.yaml
kubectl logs -n default deploy/collector > collector-before.log

-rw-r--r--  1 mayowababatola  staff   6206 Nov  7 18:05 audit-jordan.jsonl
-rw-r--r--  1 mayowababatola  staff  12414 Nov  7 18:05 audit-reporting.jsonl
-rw-r--r--  1 mayowababatola  staff    236 Nov  7 18:05 collector-before.log
-rw-r--r--  1 mayowababatola  staff  18470 Nov  7 18:05 objects-before.yaml
-rw-r--r--  1 mayowababatola  staff   1690 Nov  7 18:05 rbac-before.yaml

10.244.1.108 - - [07/Nov/2026 23:05:00] "GET /sync?d=[redacted] HTTP/1.1" 404 -

The collector's log has the key itself (redacted here), sent as part of the address. The collector answered 404, "not found", and that didn't matter: the request, with the key in it, had already arrived and been logged. An error response doesn't mean nothing got out. In a real attack, the collector would be outside the cluster, and this log would be the attacker's.

This took seconds. When something is actively leaking, a quick snapshot now and the rest after containment is the usual compromise.

5. Containment: the person

Remove Jordan's access:

$ kubectl delete rolebinding contractor-jordan-edit -n default
rolebinding.rbac.authorization.k8s.io "contractor-jordan-edit" deleted

$ kubectl --kubeconfig jordan.kubeconfig auth whoami
Groups                                              [contractors system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [X509SHA256=b428e298fb50b3efbcd3924b649ac90591da72549cc387364d41d390b544b67e]
$ kubectl --kubeconfig jordan.kubeconfig get secrets
Error from server (Forbidden): secrets is forbidden: User "jordan@contractor.example" cannot list resource "secrets" in API group "" in the namespace "default"

Jordan can't do anything now, but Kubernetes still knows exactly who Jordan is. Kubernetes has no way to revoke a client certificate: the certificate stays valid until it expires, and removing every permission is the only lever. That's why this one was issued for two hours, and why real clusters give people short-lived credentials through single sign-on instead of certificates.

Then the question that matters: is the key still leaving?

Is the key still leaving? Waiting for the next minute's run...
requests to the collector: 1 before, 2 now

Yes. Removing Jordan stopped Jordan. It didn't stop what Jordan left behind: the CronJob runs as the reporting account, which still had its access, and sent the key out again a minute later.

6. Containment: the machine

$ kubectl patch cronjob metrics-sync -n default -p '{"spec":{"suspend":true}}'
cronjob.batch/metrics-sync patched

$ kubectl delete rolebinding reporting-read-secrets -n default
rolebinding.rbac.authorization.k8s.io "reporting-read-secrets" deleted

$ kubectl auth can-i get secrets -n default --as=system:serviceaccount:default:reporting
no

• Suspending the CronJob stops new runs without deleting it, so it's still there as evidence.
• Removing the reporting account's access stops anything else using that account's tokens, including anything not found yet.
• The real reporting job is broken too, until recovery. That's a trade-off made on purpose: a broken report is better than a leaking key. Decisions like this are why incidents need someone in charge, who can make them quickly.

7. The rest of the evidence

-rw-r--r--  1 mayowababatola  staff  67076 Nov  7 18:06 audit-namespace.jsonl
-rw-r--r--  1 mayowababatola  staff    411 Nov  7 18:06 collector.log
-rw-r--r--  1 mayowababatola  staff   5183 Nov  7 18:06 csr-jordan.yaml
-rw-r--r--  1 mayowababatola  staff      0 Nov  7 18:06 metrics-sync-29901545-7kxbw.log
-rw-r--r--  1 mayowababatola  staff      0 Nov  7 18:06 metrics-sync-29901546-ng6pq.log
...
fab895d9b745268b  audit-jordan.jsonl
72b3be035a8ffde4  audit-namespace.jsonl
bd23d2383cfc4143  audit-reporting.jsonl
25bf2f824cc1bf6e  collector-before.log
39469c3bfa7e16ee  collector.log
385c50c52516c6bb  csr-jordan.yaml
e3b0c44298fc1c14  metrics-sync-29901545-7kxbw.log
e3b0c44298fc1c14  metrics-sync-29901546-ng6pq.log
9010760ebcde2c11  objects-before.yaml
6527843870b298c0  rbac-before.yaml

• Every file hashed with SHA-256 the moment it was saved (the first 16 characters shown). If anyone later asks whether the evidence was changed, the hashes answer. That's the start of a chain of custody: who collected what, when, and proof it hasn't changed since.
• The two job pods' logs are empty: curl -s printed nothing. Their identical hashes, e3b0c442..., are the SHA-256 of an empty file. The audit log and the collector's log are what show what they did.
• collector.log grew from 236 to 411 bytes: the second request, the one after Jordan was removed.
• The evidence contains the key. Evidence is as sensitive as what it's about; it's kept somewhere only the investigators can read.

8. Eradication

What to remove comes from the audit log, not from guessing:

$ jq -r 'select(.verb=="create") | .objectRef.resource + "/" + .objectRef.name' audit-jordan.jsonl | sort -u
cronjobs/metrics-sync
deployments/collector
selfsubjectreviews/
services/collector

$ kubectl delete cronjob metrics-sync -n default; kubectl delete deployment,service collector -n default
cronjob.batch "metrics-sync" deleted
deployment.apps "collector" deleted
service "collector" deleted

Three objects, all removed. (selfsubjectreviews isn't an object; it's the auth whoami question.) Deleting the CronJob also deleted its Jobs and their pods. This only works because Lesson 10's policy records every create. Anything Jordan did that the policy doesn't record (Lesson 10's blind spot was reads) would need other evidence, or a comparison against a baseline, Lesson 4's other half.

9. Recovery

The key left the cluster, so protecting it now is pointless. It's changed (in real life, after revoking the old one with the payment provider, so it stops working everywhere):

key fingerprint: 327d3cedfeb3 before, 91127b724b97 now

$ kubectl create rolebinding reporting-read-secrets -n default --role=read-secrets --serviceaccount=default:reporting
rolebinding.rbac.authorization.k8s.io/reporting-read-secrets created

(A fingerprint is a hash of the key, so the lesson can show it changed without showing the key.) Then the real reporting job's access back, now that nothing else uses its account.

10. Is it over?

Not "we think so": checked, from the same data that found it.

Secret requests by Jordan or reporting, last 3 minutes:
0
Audit alerts still firing:
9

At 23:15 UTC, no requests from either account. The alerts were still firing, which looked wrong for a moment: the last attack event was at 23:06:45. A few minutes later, at 23:17, the list of firing audit alerts was empty. Loki's rules look at a 5-minute window, and it takes another evaluation or two for "nothing in the window" to reach Alertmanager as resolved. An alert clearing isn't instant, and an alert still firing doesn't always mean the incident is.

11. The timeline

From the audit log, with the reporting account's repeats and the program column removed:

23:04:48  kubernetes-admin           create  default/serviceaccounts/reporting            201
23:04:49  kubernetes-admin           create  default/roles/read-secrets                   201
23:04:49  kubernetes-admin           create  default/rolebindings/reporting-read-secrets  201
23:04:49  kubernetes-admin           create  default/secrets/payments-api-key             201
23:04:49  kubernetes-admin           list    default/secrets/                             200
23:04:49  kubernetes-admin           create  default/rolebindings/contractor-jordan-edit  201
23:04:49  jordan@contractor.example  create  -/selfsubjectreviews/                        201
23:04:49  jordan@contractor.example  list    default/secrets/                             200
23:04:49  jordan@contractor.example  get     default/secrets/payments-api-key             200
23:04:49  jordan@contractor.example  create  default/deployments/collector                201
23:04:49  jordan@contractor.example  create  default/services/collector                   201
23:04:49  jordan@contractor.example  create  default/cronjobs/metrics-sync                201
23:05:30  kubernetes-admin           delete  default/rolebindings/contractor-jordan-edit  200
23:05:30  jordan@contractor.example  list    default/secrets/                             403
23:06:45  kubernetes-admin           patch   default/cronjobs/metrics-sync                200
23:06:45  kubernetes-admin           delete  default/rolebindings/reporting-read-secrets  200

(The full timeline also shows Lesson 10's tests at 22:44 and 22:53, the trap from triage.) The numbers a write-up records:

A script responded here, so these are faster than any person would be. A real team would count minutes to hours. The shape is what matters: detection was fast; containment was in the wrong order the first time.

12. Lessons learned

The write-up after an incident is a post-incident review (or postmortem). Good ones are blameless: they ask what made it possible, not who to blame, because people who fear blame hide what happened, and the next incident is worse. For this one:

What went well. Alerts in 31 seconds, naming the accounts. The audit log had every step, down to the pod. Evidence was saved and hashed before anything changed.

What made it possible:

• A credential that leaked. How Jordan's kubeconfig got out is the first question, and the audit log can't answer it.
• edit can run pods as any service account in its namespace. Jordan's role couldn't create role bindings, but it could start a pod as the reporting account, and so borrow its access. That access outlived Jordan's. This is a well-known escalation path in Kubernetes.
• The payment key was readable by a role shared with a reporting job. One Secret per job, readable only by that job, would have kept it out of reach.
• Nothing stopped the job connecting out. An egress NetworkPolicy (Lesson 9's other direction) would have blocked the collector.

What to change, each with an owner and a date in a real review: short-lived, single-sign-on credentials for people, not certificates; contractors' access only to what they need, never edit where Secrets are; one Secret per workload; egress policies; and a runbook that says find and stop what was left behind, not just who left it.

And what went wrong in the response itself, which belongs in the review too: the setup script's own read set off an alert, the triage nearly mistook an earlier test for the start of the attack, and one check failed because an old copy of the response script ran. (The "is it over?" check was redone by hand.)

Spot the problem

Work out what's wrong before you read the explanation.

Closed at the first fix. A company's alert says a contractor's account read production Secrets late at night. The on-call engineer checks the audit log, sees the contractor's reads, removes the contractor's access, confirms the account now gets Forbidden, and closes the incident with "access revoked, no further activity from the account". Three weeks later, the payment provider reports fraud with the company's key. How, if the account was locked out? The account was locked out; what it left behind wasn't. In this lesson, removing Jordan's access took 41 seconds, and the key left again a minute later, because Jordan had created a CronJob running as a different account. "No further activity from the account" was true, and beside the point. Attackers who get in leave a way to stay (persistence): a scheduled job, a new service account, a key added somewhere, a changed image. Three fixes: after removing the person, look for what they created (every create in the audit log, and a comparison against a baseline); check that the harm has stopped, not just the actor (here, the collector's request count, and "no Secret requests from either account"); and change any secret they could have read, whatever else you find, because once a key has left, only changing it makes it safe.

The end of the track

That's Security for engineers. Looking back at what Lesson 1 found:

• nginx running as root: in staging, non-root, read-only, with no capabilities and no token (Lesson 9).
• Alloy and Grafana reading every Secret: cut to what they use (Lesson 9).
• Secrets not encrypted at rest: encrypted, with the catch about old copies (Lesson 5).
• No audit log: on, in Loki, with alerts that reached Slack in 31 seconds (Lesson 10).
• Any pod reaching anything: staging's bookshop reachable only from monitoring (Lesson 9).

And around them: a threat model to decide what matters (Lesson 2), a hardened and audited server (Lessons 3 and 4), images scanned and a scanner that was itself attacked (Lesson 6), the OWASP Top 10 tried for real (Lesson 7), and TLS that renews but doesn't always reload (Lesson 8). Production still has most of Lesson 1's findings: the same pull requests, with production's values, are where to go next.

Check yourself

Key terms

• Incident: an event that threatens the confidentiality, integrity or availability of a system.
• Triage: deciding what an incident is, how serious, and whether it's ongoing.
• Containment: stopping an incident getting worse.
• Eradication: removing what an attacker left.
• Persistence: what an attacker leaves to keep access, such as a scheduled job or a new account.
• Chain of custody: a record of who collected and handled evidence, and proof it hasn't changed.
• Indicator of compromise (IOC): something that shows an attack happened, like a CronJob called metrics-sync sending to collector.
• Runbook: written steps for handling a kind of incident.
• Post-incident review: the write-up after an incident: timeline, causes, changes. Blameless when it's done well.
• Incident commander: the person who makes decisions during an incident.

Go deeper

• NIST SP 800-61 Revision 3 (opens in a new tab), Incident Response Recommendations and Considerations for Cybersecurity Risk Management.
• The Kubernetes documentation's security checklist (opens in a new tab), for what to have in place before an incident.
• Google's SRE book on postmortem culture (opens in a new tab), for blameless reviews.
• MITRE ATT&CK for Containers (opens in a new tab), the known techniques, including persistence by scheduled jobs.
• Try: add an egress NetworkPolicy to default that allows only DNS and the API server, then run the incident again. What happens to the collector's requests, and what still shows up in the audit log?
• Try: write this incident up as a one-page review, from the timeline and evidence folder, before you delete ~/Apps/incident-lab.