An incident, from alert to fix

What an incident is

An incident is anything that hurts the people using a service, or is about to, and needs someone now. Every one goes through the same stages, whatever the cause:

1. Detect: something notices. Ideally an alert, not a customer.
2. Triage: what's broken, how badly, for whom, and what's the likely cause.
3. Mitigate: stop the harm, the fastest safe way. Often a revert, and not yet a real fix.
4. Fix: the real correction, without the pressure.
5. Learn: write it up, so the same thing doesn't happen again, or is caught sooner.

The first three are fast and stressful; the last two are where the value is. This lesson does all five, with a real outage.

The change

A tidy-up to the bookshop chart. Its Service picked its pods by one label, app: bookshop. Kubernetes recommends a set of standard labels, app.kubernetes.io/name and friends, so the change moves the Service onto them, through a new helper:

--- a/charts/bookshop/templates/_helpers.tpl
+++ b/charts/bookshop/templates/_helpers.tpl
@@ -6,3 +6,9 @@ app.kubernetes.io/instance: {{ .Release.Name }}
 app.kubernetes.io/version: {{ .Values.shop.version | quote }}
 helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
 {{- end }}
+
+{{/* The labels the Service picks its pods by: the standard Kubernetes names. */}}
+{{- define "bookshop.selectorLabels" -}}
+app.kubernetes.io/name: {{ .Chart.Name }}
+app.kubernetes.io/component: web
+{{- end }}
--- a/charts/bookshop/templates/service.yaml
+++ b/charts/bookshop/templates/service.yaml
@@ -7,7 +7,7 @@ metadata:
     {{- include "bookshop.labels" . | nindent 4 }}
 spec:
   selector:
-    app: {{ .Chart.Name }}
+    {{- include "bookshop.selectorLabels" . | nindent 4 }}
   ports:
     - name: http
       port: 80

It looks reasonable, and it passed the checks a reviewer would run:

$ helm lint charts/bookshop | tail -1
1 chart(s) linted, 0 chart(s) failed

$ helm template bookshop charts/bookshop | sed -n '/^kind: Service$/,/^---/p' | sed -n '/selector:/,/ports:/p'
  selector:
    app.kubernetes.io/name: bookshop
    app.kubernetes.io/component: web
  ports:

• helm lint checks that a chart is well formed: its files, its YAML, its templates rendering.
• helm template shows what it renders to. The selector renders exactly as written.

Both right, and both beside the point: neither knows what labels the pods have. Merged at 21:17:56, as #73.

Before it, a normal evening, for comparison later:

bookshop Synced Healthy
bookshop-staging Synced Healthy

1  instance=http://bookshop.bookshop-staging.svc/ job=bookshop-up namespace=monitoring
1  instance=http://bookshop.bookshop.svc/ job=bookshop-up namespace=monitoring

And who was asking the shops for pages, from Loki, in 2 minutes:

24  bookshop  kube-probe/1.35
12  bookshop  Blackbox-Exporter/0.29.1
8  bookshop  NGINX-Prometheus-Exporter/v0.0.0-SNAPSHOT-9956da7
4  bookshop  null
12  bookshop-staging  kube-probe/1.35
8  bookshop-staging  Blackbox-Exporter/0.29.1
4  bookshop-staging  NGINX-Prometheus-Exporter/v0.0.0-SNAPSHOT-9956da7

The query was Lesson 9's, for both namespaces at once: sum by (namespace, agent) (count_over_time({app="bookshop", container="nginx"} | pattern ... [2m])). Production has two pods, so twice staging's probes; null is four lines that didn't fit the pattern, so had no agent to count.

The alert

Argo CD synced production 4 seconds after the merge, and staging on its own schedule, almost 3 minutes later. From the alert receiver's log, with --timestamps:

kubectl logs -n monitoring deploy/alert-receiver --since-time=2026-11-06T02:17:56Z --timestamps \
  | grep ' {' | while read -r ts json; do echo "${ts:11:8} UTC $(jq -c '[.status, .groupLabels.instance]' <<<"$json")"; done

• --timestamps puts the time Kubernetes received each line in front of it.
• while read -r ts json splits each line into the time and the notification; ${ts:11:8} keeps just HH:MM:SS.

02:20:41 UTC ["firing","http://bookshop.bookshop.svc/"]
02:23:11 UTC ["firing","http://bookshop.bookshop-staging.svc/"]

Firing: BookshopDown, for production at 21:20:41, 2 minutes 41 seconds after it synced, and the same for staging at 21:23:11. Each the same message in Slack. The 2 minutes 41 seconds is everything from the earlier lessons in a row: the next blackbox check (every 30 seconds, Lesson 5), BookshopDown's for: 1m (Lesson 7), and Alertmanager's group_wait: 30s (Lesson 8).

Two alerts, two messages, and both shops. That's a clue already: something they share.

Triage

The on-call person's job now: find what's broken, and the likely cause, fast. Not every possible question; the right ones, in an order where each answer rules something out. Here, seven, each answered by a tool from this track.

1. Is the monitoring working? An alert is only as good as what sent it. The heartbeat, from Lesson 10:

{"status":"grace","last_ping":"2026-11-06T02:27:06+00:00"}

grace isn't down: the last ping was under 2 minutes before, and pings come every 2 minutes (Lesson 10's finding), so the check is in its grace time. The monitoring works; the alert is real.

2. What does Alertmanager have?

active  BookshopDown http://bookshop.bookshop-staging.svc/
active  BookshopDown http://bookshop.bookshop.svc/
suppressed  BookshopPageWrong http://bookshop.bookshop-staging.svc/  inhibitedBy=1
suppressed  BookshopPageWrong http://bookshop.bookshop.svc/  inhibitedBy=1

Four alerts, two messages: Lesson 8's inhibit rule at work. The page check fails because the shop doesn't answer at all. One problem, per shop.

3. Why are the checks failing? The blackbox exporter explains each check it makes, with debug=true (Lesson 5):

curl -s 'http://localhost:9115/probe?module=http_2xx&target=http://bookshop.bookshop.svc/&debug=true' | grep -i -m3 -e 'error' -e 'refused' -e 'probe failed'

level=ERROR source=http.go:584 msg="Error for HTTP request" module=http_2xx target=http://bookshop.bookshop.svc/ err="Get \"http://10.96.111.138/\": dial tcp 10.96.111.138:80: connect: connection refused"
level=ERROR source=handler.go:140 msg="Probe failed" module=http_2xx target=http://bookshop.bookshop.svc/ duration_seconds=0.002521625

connection refused, in 2.5 milliseconds. Not slow, not a wrong page, not a timeout: nothing listening at the Service's address, 10.96.111.138, at all. On Kubernetes, the usual reason is a Service with nothing behind it.

4. What does Argo CD say?

bookshop Synced Healthy
bookshop-staging Synced Healthy

Synced Healthy. The cluster matches Git, so whatever is broken is in Git. And "Healthy" only means each resource is fine by its own measure: a Deployment with its pods ready, a Service that exists. Nothing in Argo CD asks whether the Service reaches the pods. That's what the outside checks are for.

5. Are the pods all right?

NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
bookshop-65d8d9ff66-2krfn   2/2     Running   0          9h    10.244.1.29   gitops-worker   <none>           <none>
bookshop-65d8d9ff66-6dwvg   2/2     Running   0          9h    10.244.1.30   gitops-worker   <none>           <none>

Running, ready, no restarts, untouched for 9 hours. The shop itself is fine.

6. Who can still reach them? The same Loki query as before:

23  bookshop  kube-probe/1.35
12  bookshop-staging  kube-probe/1.35

Only the kubelet's probe. The blackbox exporter's requests are gone, and so are the nginx exporter's, since Prometheus reaches the exporter through the Service too. The kubelet still gets through because it goes to each pod's own address, not the Service's. So the pods answer, and the traffic stops at the Service.

7. What does the Service point at?

kubectl get endpointslices -n bookshop -l kubernetes.io/service-name=bookshop
kubectl get service bookshop -n bookshop -o jsonpath='{.spec.selector}'; echo
kubectl get pods -n bookshop -l app.kubernetes.io/name=bookshop,app.kubernetes.io/component=web
kubectl get pods -n bookshop -L app,app.kubernetes.io/name,app.kubernetes.io/component

• An EndpointSlice is Kubernetes's list of the pod addresses behind a Service, kept up to date from the Service's selector.
• -l lists pods matching labels, the same way the Service's selector does; -L shows labels as columns.

NAME             ADDRESSTYPE   PORTS     ENDPOINTS   AGE
bookshop-w7xcn   IPv4          <unset>   <unset>     9h

{"app.kubernetes.io/component":"web","app.kubernetes.io/name":"bookshop"}

No resources found in bookshop namespace.

NAME                        READY   STATUS    RESTARTS   AGE   APP        NAME       COMPONENT
bookshop-65d8d9ff66-2krfn   2/2     Running   0          9h    bookshop   bookshop
bookshop-65d8d9ff66-6dwvg   2/2     Running   0          9h    bookshop   bookshop

The cause. The Service wants pods with component: web. The pods have app and name, and an empty COMPONENT column. No pod matches, so the Service has no endpoints, and every connection to it is refused. And when it changed:

$ argocd app history bookshop | tail -3
1       2026-11-05 11:56:37 -0500 EST  main (b2e5b74)
2       2026-11-05 11:58:12 -0500 EST  main (2486822)
3       2026-11-05 21:18:00 -0500 EST  main (ffeb873)

$ git log --oneline -3 -- charts/bookshop
ffeb873 Bookshop: select pods by the standard labels (#73)
2b9bbdb Revert "Staging: no pods (#64)" (#65)
c9daa85 Staging: no pods (#64)

Synced at 21:18:00, from ffeb873, the tidy-up. git log -- charts/bookshop lists only commits that touched the chart: the last change to it, minutes before the alert.

Mitigate: revert first

There are two ways out. Fix forward: add component: web to the pods' labels, or take it out of the selector. Roll back: revert the commit, and go back to what worked an hour ago. In an incident, roll back first, almost always:

• It's known to work. The old selector ran for 9 hours. A fix forward is a new change, written under pressure, and untested.
• It's fast. One command, and Argo CD does the rest.
• The real fix can wait. Once the shop is back, there's time to do the change properly, reviewed, and tried on staging.

merged: 4599223 Revert "Bookshop: select pods by the standard labels (#73)" (#74)

Argo CD synced both shops within 22 seconds of the merge, and the endpoints came straight back:

NAME             ADDRESSTYPE   PORTS     ENDPOINTS                 AGE
bookshop-w7xcn   IPv4          80,9113   10.244.1.29,10.244.1.30   9h

Notice AGE 9h: the same EndpointSlice, filled again, not a new one. The pods never moved; only the Service's selector did.

Verify

Mitigated isn't done until the evidence says so. Lesson 10's lesson: a revert is a change to what you want, not proof of what you have.

["firing","http://bookshop.bookshop.svc/","BookshopDown"]
["firing","http://bookshop.bookshop-staging.svc/","BookshopDown"]
["resolved","http://bookshop.bookshop-staging.svc/","BookshopDown"]
["resolved","http://bookshop.bookshop.svc/","BookshopDown"]

1  instance=http://bookshop.bookshop-staging.svc/ job=bookshop-up namespace=monitoring
1  instance=http://bookshop.bookshop.svc/ job=bookshop-up namespace=monitoring

Resolved in Slack for both shops, at 21:30:11 and 21:30:41; both checks passing; and Alertmanager's Bookshop alerts empty. Four messages in all: two firing, two resolved. And how long each shop was down, from Prometheus:

curl -s -G http://localhost:9090/api/v1/query --data-urlencode \
  'query=sum by (instance) (count_over_time((probe_success{job="bookshop-up", instance=~".*svc/$"} == 0)[30m:30s])) * 30'

• (probe_success ... == 0)[30m:30s] is a subquery: the inner query evaluated every 30 seconds for the last 30 minutes, keeping only the failed checks.
• count_over_time counts them, and * 30 turns 30-second steps into seconds.

510  instance=http://bookshop.bookshop-staging.svc/
660  instance=http://bookshop.bookshop.svc/

Production down for 660 seconds, 11 minutes; staging for 510, because it got the change almost 3 minutes later.

The 8 minutes nobody was looking

The first alert arrived at 21:20:41. The investigation started at 21:28:53. For 8 minutes, production was down, Slack had said so, and nobody did anything.

The reason is honest and ordinary. The script running this lesson stalled: before the run, a git pull from a tool that couldn't delete files had left a lock file in the repository, .git/ORIG_HEAD.lock. Git refuses to work while a lock file exists, in case another Git command is part-way through something. So the script's own merge step half-failed: the change reached GitHub, and the cluster, but the laptop's copy of main didn't update, and the script sat waiting for a commit that would never sync. The outage was real; the tooling meant to handle it was stuck.

That's what real incidents are like. The alert worked; the response didn't start. In a real team, the alert pages a person whose phone rings until they acknowledge it, and if they don't within a few minutes, it pages someone else. A Slack message on its own doesn't do that.

The write-up

After the fix, the most valuable part: a postmortem, written down, shared, and blameless. Blameless means it asks what about the system let this happen, never who did it. Someone made a reasonable change, and every check said yes. Blame teaches people to hide mistakes; a blameless write-up finds the checks that were missing. Here's this one.

Action items, each a change to the system, each with an owner and a date in a real team:

1. Check selectors against pods before merging. A check in the pull request that renders the chart and fails if any Service's selector doesn't match its Deployment's pod labels. A few lines of yq, or a policy tool such as Conftest or Kyverno.
2. Send chart changes to staging first. Give the chart a version, and point staging's Application at the new version before production's, so template changes get the staging-first treatment values already have.
3. Alert on a Service with no endpoints. kube-state-metrics reports the addresses behind each Service: a rule for "zero ready addresses for 1 minute" names the cause directly, instead of a symptom.
4. Page for critical alerts. critical to a pager that rings until someone acknowledges, with a second person if nobody does. Slack for warning.

(The investigation here took seconds because a script ran it. A person takes minutes: that's why the order of questions matters, and why the first one is always "is the monitoring itself working?")

Spot the problem

Work out what's wrong before you read the explanation.

The false alarm that wasn't. It's 2 a.m. BookshopDown fires. The on-call engineer opens Argo CD: Synced Healthy. kubectl get pods: all Running, all ready. Lesson 8's silence form is right there, so they silence BookshopDown for 8 hours, "pods fine, false alarm, will check in the morning", and go back to sleep. In the morning, the shop has been down all night. What went wrong? They checked the wrong things, and trusted green over the alert. Argo CD's health and the pods' status say each piece is fine on its own; this lesson's outage had both, all the way through. The blackbox check is the only one asking the question that matters: does the shop answer, from outside? The habits that prevent it: never silence an alert you can't explain (the check said connection refused; that's not nothing); confirm the monitoring first, then believe the alert (the heartbeat was up, so the check was real); and follow the request's path (address, Service, endpoints, pod) until it breaks.

Where you are now

That's the Monitoring track. You started with a cluster nobody was watching. Now it has metrics with labels, PromQL to ask questions of them, the bookshop's own numbers, checks from outside, dashboards in Git, alert rules tested before they merge, Alertmanager routing to Slack, logs in Loki, a heartbeat that catches the monitoring itself failing, and the habit of using them in order, under pressure. Every piece of it is in Git, and could be rebuilt from there.

Next comes a Monitoring cheatsheet: the commands from all eleven lessons, on one page.

Check yourself

Key terms

• Incident: anything hurting a service's users, or about to, that needs someone now.
• Triage: working out what's broken, how badly, and the likely cause, in order.
• Mitigation: stopping the harm the fastest safe way, often a revert, before the real fix.
• Roll back and fix forward: going back to the last version that worked, or correcting the new one in place.
• Endpoints: the pod addresses behind a Service, kept in EndpointSlices from its selector.
• Acknowledge: an on-call person confirming they've seen a page, which stops it escalating.
• Postmortem: the written account of an incident: impact, timeline, causes, and action items.
• Blameless: looking at what in the system let something happen, never at who.

Go deeper

• Google's Site Reliability Engineering book, free online: Postmortem Culture: Learning from Failure (opens in a new tab) and Managing Incidents (opens in a new tab).
• PagerDuty's open incident response guide (opens in a new tab), the roles and steps many teams copy.
• The Kubernetes documentation on EndpointSlices (opens in a new tab) and its recommended labels (opens in a new tab).
• Try: do action item 1. Write a check that renders the chart with helm template, pulls out the Service's selector and the Deployment's pod labels with yq, and fails if any selector label is missing from the pods. Run it on the tidy-up's commit, and see it fail.
• Try: the real fix. Add app.kubernetes.io/component: web to the pods' labels first, in one pull request, then change the Service's selector in a second, and watch nothing break.