Alert rules

What an alert rule is

Every check so far needed someone to look: a query, a dashboard. An alert rule looks for you. It's a PromQL query and a time:

- alert: BookshopDown
  expr: probe_success{job="bookshop-up", instance=~".*svc/$"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "The bookshop isn't answering"
    description: "{{ $labels.instance }} has failed its check for over a minute."

• alert: its name.
• expr: a query. Every series it returns is a problem. Lesson 3's up == 0 pattern: the comparison leaves only what's wrong.
• for: how long it must keep returning that series before the alert fires. Until then, it's pending. A blip doesn't wake anyone.
• labels: added to the alert. severity is the one everyone uses, and Lesson 8 routes on it.
• annotations: the words a person reads. {{ $labels.instance }} is filled in from the series, so the alert says which page.

Prometheus checks every rule every 30 seconds, the evaluation_interval from Lesson 2. That's all it does: it doesn't send messages. Alertmanager does, in Lesson 8.

Quiet first

An alert that fires for something nobody needs to fix teaches people to ignore alerts. So before adding any, a look at what was already firing on a cluster where nothing is wrong:

$ alerts
   1 firing critical etcdInsufficientMembers kube-system
   1 firing none Watchdog
   1 firing warning etcdMembersDown kube-system
   1 firing warning KubeControllerManagerInstanceUnreachable 172.18.0.2:10257
   1 firing warning KubeProxyInstanceUnreachable 172.18.0.2:10249
   1 firing warning KubeProxyInstanceUnreachable 172.18.0.3:10249
   1 firing warning KubeSchedulerInstanceUnreachable 172.18.0.2:10259
   1 firing warning NodeClockNotSynchronising 172.18.0.2:9100
   1 firing warning NodeClockNotSynchronising 172.18.0.3:9100
   4 firing warning TargetDown kube-system

(alerts is a short function that counts Prometheus's alerts by state, severity, name and target.) Thirteen alerts, one of them critical, and Lesson 1 explained every one: kind keeps four control-plane parts where Prometheus can't reach them, and kind's nodes don't sync their clocks. None of it is something to fix. The chart has a setting for each. Its own values file says so:

2121:kubeControllerManager:
2465:kubeEtcd:
2602:kubeScheduler:
2762:kubeProxy:

defaultRules:
  rules:
    etcd: true
    kubeControllerManager: true
    kubeProxy: true
    kubeSchedulerAlerting: true
    ...
  disabled: {}

(From helm show values prometheus-community/kube-prometheus-stack --version 92.18.0, which prints every setting a chart version has. Worth reading before changing one.) Each part has an enabled setting, for whether to scrape it; defaultRules.rules switches whole groups of rules; defaultRules.disabled switches off single alerts by name. In the monitoring Application's values:

        # kind keeps these control-plane parts unreachable (Lesson 1), so their targets
        # are always down and their alerts always fire. Stop scraping them (Lesson 7).
        kubeEtcd:
          enabled: false
        kubeControllerManager:
          enabled: false
        kubeScheduler:
          enabled: false
        kubeProxy:
          enabled: false
        defaultRules:
          rules:
            # And drop the alert rules about them.
            etcd: false
            kubeControllerManager: false
            kubeProxy: false
            kubeSchedulerAlerting: false
          disabled:
            # kind's nodes don't sync their clocks; nothing to act on, here.
            NodeClockNotSynchronising: true

Merged as #50. Three minutes later:

$ curl -s http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | .health' | sort | uniq -c
  26 up

$ curl -s http://localhost:9090/api/v1/rules | jq '[.data.groups[].rules[] | select(.type=="alerting")] | length'
133

$ alerts
   1 firing none Watchdog
   4 firing warning TargetDown kube-system
   1 pending info CPUThrottlingHigh 172.18.0.3:10250

Every target up, 155 rules down to 133, and most of the noise gone. But two things worth a closer look. The four TargetDown alerts were still firing, for targets that no longer existed:

firing job=kube-scheduler service=monitoring-kube-prometheus-kube-scheduler namespace=kube-system since 19:36:45
firing job=kube-proxy service=monitoring-kube-prometheus-kube-proxy namespace=kube-system since 19:36:45
firing job=kube-etcd service=monitoring-kube-prometheus-kube-etcd namespace=kube-system since 20:10:15
firing job=kube-controller-manager service=monitoring-kube-prometheus-kube-controller-manager namespace=kube-system since 20:10:15

A couple of minutes later, they'd cleared by themselves. TargetDown works on the up series, and the last value of a series that stops being scraped doesn't vanish the instant its target goes. Most likely, the old up = 0 series lingered until Prometheus stopped treating them as current. Either way: after removing the cause, give an alert a few minutes before deciding the fix didn't work.

And one alert that was new: CPUThrottlingHigh, pending, severity info:

pending kube-system/kindnet-bn9n5 container=kindnet-cni 35.1% throttling of CPU in namespace kube-system for container kindnet-cni in pod kindnet-bn9n5 on cluster .

kind's network plugin, kindnet, hitting its CPU limit 35% of the time. That's real, and it's information, not a problem: the network works. info is the chart's way of saying "worth knowing, not worth waking anyone for". Lesson 8 sends info alerts nowhere, on purpose.

Rules for the bookshop

Four rules, in a new file, apps/monitoring/bookshop-rules.yaml, as a PrometheusRule, another of Lesson 1's resource kinds. First, the label check that Lessons 4 and 5 taught:

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.ruleSelector}'
{"matchLabels":{"release":"monitoring"}}

The same rule again: only PrometheusRules labelled release: monitoring. The file:

# Alert rules for the bookshop (Monitoring track, Lesson 7).
# The release label is what this cluster's Prometheus looks for, as with ServiceMonitors.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: bookshop
  labels:
    release: monitoring
spec:
  groups:
    - name: bookshop
      rules:
        - alert: BookshopDown
          # The blackbox check (Lesson 5), for each environment's page. Not the missing-page target.
          expr: probe_success{job="bookshop-up", instance=~".*svc/$"} == 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "The bookshop isn't answering"
            description: "{{ $labels.instance }} has failed its check for over a minute."
        - alert: BookshopPageWrong
          # The banner check: the page answers, but doesn't say what it should.
          expr: probe_success{job="bookshop-banner"} == 0
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "The bookshop's page is wrong"
            description: "{{ $labels.instance }} is answering without the banner it should have."
        - alert: BookshopPodsMissing
          # kube-state-metrics (Lesson 2): fewer pods available than the Deployment wants.
          expr: kube_deployment_spec_replicas{deployment="bookshop"} > kube_deployment_status_replicas_available{deployment="bookshop"}
          for: 3m
          labels:
            severity: warning
          annotations:
            summary: "The bookshop is short of pods"
            description: "{{ $labels.namespace }}/{{ $labels.deployment }} has fewer pods available than it wants."
        - alert: BookshopChecksMissing
          # No check results at all: the blackbox exporter, or the Probe, has gone (Lesson 3's lesson).
          expr: absent(probe_success{job="bookshop-up"})
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "The bookshop isn't being checked"
            description: "There are no blackbox results for the bookshop, so BookshopDown can't fire."

Why each one, and why its numbers:

• BookshopDown, critical, 1 minute: the shop isn't answering, the thing customers notice first. instance=~".*svc/$" leaves out Lesson 5's missing-page target, which fails on purpose. Without that filter, this alert would fire forever.
• BookshopPageWrong, warning, 2 minutes: Lesson 5's banner check. Wrong is bad, but it isn't down.
• BookshopPodsMissing, warning, 3 minutes: Lesson 3's wanted-against-available comparison. Long enough that a normal rolling update, which briefly has a pod starting, never sets it off.
• BookshopChecksMissing, warning, 5 minutes: the answer to Lesson 3's lesson. If the blackbox exporter or the Probe is gone, probe_success doesn't exist, so probe_success == 0 returns nothing, and BookshopDown can never fire. absent() returns a series exactly when the one inside doesn't exist: it turns missing data into an alert.

Testing rules before merging

A rule with a typo in the metric name is valid, and silent forever: Lesson 3's misspelled MemTotal. promtool, which comes with Prometheus, checks rules two ways. promtool check rules reads the file and checks every expression is valid PromQL. promtool test rules goes further: it feeds the rules made-up series and checks they fire when they should. A test file:

rule_files: [rules.yaml]
evaluation_interval: 30s
tests:
  - interval: 30s
    input_series:
      - series: 'kube_deployment_spec_replicas{namespace="bookshop", deployment="bookshop"}'
        values: '2x20'
      - series: 'kube_deployment_status_replicas_available{namespace="bookshop", deployment="bookshop"}'
        values: '2 2 1x18'
      - series: 'probe_success{job="bookshop-up", instance="http://bookshop.bookshop.svc/"}'
        values: '1 1 0x18'
    alert_rule_test:
      - eval_time: 2m
        alertname: BookshopPodsMissing
        exp_alerts: []
      - eval_time: 5m
        alertname: BookshopPodsMissing
        exp_alerts:
          - exp_labels: { severity: warning, namespace: bookshop, deployment: bookshop }
            exp_annotations:
              summary: "The bookshop is short of pods"
              description: "bookshop/bookshop has fewer pods available than it wants."
      - eval_time: 90s
        alertname: BookshopDown
        exp_alerts: []
      - eval_time: 3m
        alertname: BookshopDown
        exp_alerts:
          - exp_labels: { severity: critical, job: bookshop-up, instance: "http://bookshop.bookshop.svc/" }
            exp_annotations:
              summary: "The bookshop isn't answering"
              description: "http://bookshop.bookshop.svc/ has failed its check for over a minute."

• input_series: made-up series, one value every 30 seconds. '2x20' is 2, then 20 more 2s. '2 2 1x18' is 2, 2, then 1 from the one-minute mark: a pod goes missing.
• alert_rule_test: at each eval_time, which alerts should be firing, with exactly which labels and words.
• So: the pod goes missing at 1 minute. At 2 minutes, BookshopPodsMissing is only pending, so no firing alert. At 5 minutes, past its 3, it's firing. The same pattern for BookshopDown, with its 1 minute.

The rules go in a file with just their groups:, the part under spec:. Then:

promtool check rules rules.yaml && promtool test rules test.yaml

Checking rules.yaml
  SUCCESS: 4 rules found

  SUCCESS

Four valid rules, and every test passed. BookshopPodsMissing is the one alert in this lesson not made to fire for real, because breaking it safely is hard: a rolling update keeps old pods running until new ones are ready, as Argo CD Lesson 9 showed. That's what tests are for: the alerts you can't safely break.

One more thing before the pull request: Lesson 6's Kustomize rule. The new file has to be in kustomization.yaml's list, or Argo CD won't apply it:

 resources:
   - grafana-admin.yaml
   - bookshop-probes.yaml
+  - bookshop-rules.yaml

$ kubectl kustomize apps/monitoring | grep -e '^kind:' -e '^  name:' | paste - - | sort
kind: ConfigMap	  name: grafana-dashboard-bookshop
kind: Probe	  name: bookshop-banner
kind: Probe	  name: bookshop-up
kind: PrometheusRule	  name: bookshop
kind: SealedSecret	  name: grafana-admin

Five resources, the new one included. Merged as #51, and a minute later:

$ curl -s http://localhost:9090/api/v1/rules | jq -r '.data.groups[] | select(.name=="bookshop") | .rules[] | "\(.health) \(.state) \(.name)"'
unknown unknown BookshopDown
unknown unknown BookshopPageWrong
unknown unknown BookshopPodsMissing
unknown unknown BookshopChecksMissing

Loaded, and unknown: not evaluated yet. On the next evaluation, every 30 seconds, they become ok and inactive.

Firing on purpose: the wrong page

Lesson 5's typo banner, on staging, a third time:

merged: ba506e1 Staging: a new banner (#52)

after 0s: BookshopPageWrong inactive
after 20s: BookshopPageWrong pending
after 141s: BookshopPageWrong firing

Pending at 20 seconds: firing at 141. The banner check failed on its next run; the rule saw that on its next evaluation; and for: 2m held it at pending until the problem had lasted two minutes. The alert itself:

{
  "state": "firing",
  "activeAt": "2026-11-05T21:30:50.307312535Z",
  "labels": {
    "alertname": "BookshopPageWrong",
    "severity": "warning",
    "instance": "http://bookshop.bookshop-staging.svc/"
  },
  "annotations": {
    "description": "http://bookshop.bookshop-staging.svc/ is answering without the banner it should have.",
    "summary": "The bookshop's page is wrong"
  }
}

The annotations filled in, naming staging's address. activeAt is when it became pending. Then the revert, as in Lesson 5:

merged: 250669f Revert "Staging: a new banner (#52)" (#53)

after 0s: BookshopPageWrong firing
after 20s: BookshopPageWrong inactive

Cleared in 20 seconds. An alert has no for on the way out: one evaluation with nothing wrong, and it's over.

Firing on purpose: no pods

Now the worst kind of broken: staging with no pods at all, through Git.

-replicas: 1
+replicas: 0

merged: 0dfde81 Staging: no pods (#54)

$ status bookshop-staging; kubectl get pods -n bookshop-staging
Synced Healthy 0dfde8182b03c88
NAME                        READY   STATUS      RESTARTS   AGE
bookshop-6869cf94c9-w8rc8   0/2     Completed   0          41s

after 0s: BookshopDown inactive
after 11s: BookshopDown pending
after 71s: BookshopDown firing

Argo CD: Synced Healthy. Of course: Git asked for zero pods, and there are zero pods, the last one finishing. Healthy means "as asked", not "useful". BookshopDown: pending in 11 seconds, firing at 71, critical:

$ q 'probe_http_status_code{job="bookshop-up", instance="http://bookshop.bookshop-staging.svc/"}'
0

$ q 'nginx_up{namespace="bookshop-staging"}' | jq '.data.result | length'
0

Status code 0: not a code at all, because nothing answered. And nginx_up for staging: no series. With no pods, there's no exporter, so nginx's own metrics simply vanish. An alert on nginx_up == 0 would never have fired here. The outside check is the one that still has something to say when everything inside is gone. Then everything firing:

$ alerts
   1 firing critical BookshopDown http://bookshop.bookshop-staging.svc/
   1 firing none InfoInhibitor kube-system
   1 firing none Watchdog
   1 pending info CPUThrottlingHigh 172.18.0.3:10250
   1 pending warning BookshopPageWrong http://bookshop.bookshop-staging.svc/

One problem, two alerts. BookshopDown, and BookshopPageWrong pending behind it: a page that doesn't answer doesn't have the right banner either. During a real outage, this is how an inbox fills with twenty alerts for one cause. Lesson 8 groups and silences them. (InfoInhibitor is the chart's own: it fires whenever an info alert does, here CPUThrottlingHigh, so Alertmanager can hold info alerts back. Also Lesson 8.) And the revert:

merged: 9cac844 Revert "Staging: no pods (#54)" (#55)

after 0s: BookshopDown firing
after 10s: BookshopDown inactive

Back in 10 seconds: Watchdog, InfoInhibitor and CPUThrottlingHigh left, and nothing about the bookshop.

Spot the problem

Work out what's wrong before you read the explanation.

The alert that couldn't fire. A team writes an alert for their app's exporter: nginx_up == 0, for 2 minutes, critical. They test it by stopping nginx inside a pod, and it fires. Months later, a bad release leaves the Deployment with no running pods for half an hour. The alert never fires. What went wrong? nginx_up == 0 needs nginx_up to exist. Stopping nginx left the exporter running, reporting nginx_up 0, so the test passed. With no pods, there's no exporter, no nginx_up at all, as staging showed, and a comparison on a series that doesn't exist returns nothing. The rule had a gap exactly where the worst failure is. Two fixes, both in this lesson: check from the outside, as BookshopDown does, and cover missing data with absent(), as BookshopChecksMissing does. And when testing an alert, break it the way it will really break, not the most convenient way.

Where you are now

Prometheus watches the bookshop for you: four rules, tested before they were merged, and two of them seen firing and clearing for real. The noise from Lesson 1 is gone, and what's left you can explain. But an alert that fires in Prometheus still reaches nobody. Lesson 8 sends them somewhere: grouped, so one outage isn't twenty messages, routed by severity, and silenced while you fix things, with Slack as the destination.

Check yourself

Key terms

• Alert rule: a PromQL expression and a for time; while the expression returns a series, there's a problem.
• Pending: an alert whose expression is true, not yet for long enough.
• Firing: an alert that has been true for its whole for time.
• Severity: a label saying how urgent an alert is, such as critical, warning or info.
• Annotations: an alert's words for people, such as a summary and a description, filled in from labels.
• PrometheusRule: a resource holding alert rules, picked up by Prometheus through its ruleSelector.
• absent(): returns a series exactly when the series inside it doesn't exist.
• promtool: Prometheus's own tool for checking and testing rules.
• Runbook: a page saying what to check and do when an alert fires.

Go deeper

• The Prometheus documentation: alerting rules (opens in a new tab) and unit testing rules (opens in a new tab), and the Google SRE book's chapter on monitoring distributed systems (opens in a new tab), on symptoms, causes and paging.
• Try: add a test for BookshopChecksMissing: input series with no probe_success at all. When should it fire?
• Try: add a runbook_url annotation to BookshopDown, pointing at a page you write: what to check first when the shop isn't answering.