PromQL: asking questions

Asking from the terminal

Lesson 1's q function sends a query, encoded properly. This lesson adds a little: it prints the query, and either the error or each answer with only the labels you name:

ask(){
  echo "\$ q '$1'"
  curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=$1" | jq -r --arg l "${2:-}" '
    if .status != "success" then "error: \(.error)"
    elif (.data.result | length) == 0 then "(no results)"
    else .data.result[] | "\(.value[1])  \(.metric as $m | ($l | split(",") | map(select(. != "")) | map("\(.)=\($m[.] // "")") | join(" ")))"
    end'
}

Paste it into your terminal once. Then ask 'QUERY' label,label runs a query and shows those labels. Every answer below came from it, on the practice cluster.

Picking series

A metric's name on its own gives every series with that name. Braces narrow it down:

$ q 'up{job="kubelet"}'
1  job=kubelet instance=172.18.0.3:10250 metrics_path=/metrics/cadvisor
1  job=kubelet instance=172.18.0.3:10250 metrics_path=/metrics
1  job=kubelet instance=172.18.0.2:10250 metrics_path=/metrics/cadvisor
1  job=kubelet instance=172.18.0.3:10250 metrics_path=/metrics/probes
1  job=kubelet instance=172.18.0.2:10250 metrics_path=/metrics
1  job=kubelet instance=172.18.0.2:10250 metrics_path=/metrics/probes

Six kubelet targets: three paths on each of the two nodes, as Lesson 1 said. Four ways to match a label:

The patterns are regular expressions: .* means "anything", | means "or", and the pattern has to match the whole value. And a comparison filters by value:

$ q 'up == 0'
0  job=kube-proxy instance=172.18.0.3:10249
0  job=kube-scheduler instance=172.18.0.2:10259
0  job=kube-proxy instance=172.18.0.2:10249
0  job=kube-etcd instance=172.18.0.2:2381
0  job=kube-controller-manager instance=172.18.0.2:10257

up == 0 keeps only the series where the comparison is true: the five targets kind won't let Prometheus reach, from Lesson 1. That's what most alert rules are: a query, and a comparison, and whatever's left is what's wrong. Add bool, as up == bool 0, and instead of filtering, it gives every series a 1 for true or 0 for false.

Counters need rate

A counter's raw value is rarely the answer, as Lesson 2 showed. Three ways to turn it into one, on the same series:

$ q 'node_cpu_seconds_total{mode="idle",cpu="0"}'
192196.11  instance=172.18.0.3:9100

$ q 'rate(node_cpu_seconds_total{mode="idle",cpu="0"}[5m])'
0.9630370370369983  instance=172.18.0.3:9100

$ q 'increase(node_cpu_seconds_total{mode="idle",cpu="0"}[5m])'
288.9111111110995  instance=172.18.0.3:9100

$ q 'irate(node_cpu_seconds_total{mode="idle",cpu="0"}[5m])'
0.9620962096204964  instance=172.18.0.3:9100

(One node shown; the other was within a thousandth.)

• rate(X[5m]): how much the counter grew per second, on average, over the last 5 minutes. 0.963: CPU 0 was idle 96.3% of the time.
• increase(X[5m]): how much it grew over those 5 minutes in total. 288.9 seconds of the 300. It's the rate times the window, and easier to read when you want a count: "errors in the last hour".
• irate(X[5m]): the rate from only the last two samples in the window. Quicker to react to a spike, and jumpier. Good on a graph you're watching live; poor in an alert, where one odd sample shouldn't decide.

[5m] is the window. It needs to hold at least two samples, and a few more to be steady: with a scrape every 30 seconds, [5m] holds about ten. Use rate() unless you have a reason not to. Both also handle a counter that went back to zero when its program restarted, which a plain subtraction wouldn't.

Adding up

One CPU at a time isn't usually the question. Aggregation combines series, and by says what to keep apart:

$ q 'sum(rate(node_cpu_seconds_total{mode!="idle"}[5m]))'
0.6017385072867769

$ q 'sum by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))'
0.30022222222222295  instance=172.18.0.3:9100
0.30151628506455397  instance=172.18.0.2:9100

Everything that isn't idle, added up: 0.6 of a CPU in total, 0.3 on each node. A CPU's seconds per second, summed over its CPUs, is a count of CPUs kept busy. Which modes?

$ q 'sum by (mode) (rate(node_cpu_seconds_total{instance="172.18.0.2:9100"}[5m]))'
15.580514218413267  mode=idle
0.0031110880660142923  mode=iowait
0  mode=irq
0  mode=nice
0.045370034296042014  mode=softirq
0  mode=steal
0.07755498107421341  mode=system
0.17548018162828422  mode=user

The eight modes add up to just under 16: sixteen CPUs, each spending every second somewhere. 15.58 of them idle, 0.18 running programs (user), 0.08 running the operating system (system). And as a fraction:

$ q '1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))'
0.026101851851853985  instance=172.18.0.3:9100
0.026217861349170812  instance=172.18.0.2:9100

The average idle fraction across the 16 CPUs, taken away from 1: 2.6% busy. That's how a dashboard's "CPU used" is usually worked out. without is the other way to say it: max without (cpu) (...) combines away the cpu label and keeps the rest:

$ q 'max without (cpu) (rate(node_cpu_seconds_total{mode="user"}[5m]))'
0.012629629629629512  instance=172.18.0.3:9100 mode=user
0.012777683128272871  instance=172.18.0.2:9100 mode=user

The busiest single CPU spent 1.3% of its time on programs. sum, avg, min, max and count all take by or without.

One catch, particular to kind: both nodes are the same machine, as Lesson 2 found. 0.3 on each node is the same 0.3, counted twice. On real servers, the sum across nodes means something; here it doesn't.

Arithmetic

Two metrics, divided:

$ q '1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes'
0.6433923291488896  instance=172.18.0.3:9100
0.642034826323836  instance=172.18.0.2:9100

$ q 'node_memory_MemAvailable_bytes / 1024 / 1024 / 1024'
2.7627830505371094  instance=172.18.0.3:9100

64% of memory in use, 2.76 GB available. Dividing one metric by another works series by series: each MemAvailable series is matched with the MemTotal series that has exactly the same labels, the same instance, job and the rest. Dividing by a plain number applies to every series.

The biggest, and the top five

$ q 'sort_desc(sum by (namespace) (container_memory_working_set_bytes{container!=""}))'
1519792128  namespace=kube-system
1342099456  namespace=monitoring
1022431232  namespace=argocd
42921984  namespace=local-path-storage
28663808  namespace=bookshop
25853952  namespace=guestbook
13697024  namespace=bookshop-staging

$ q 'topk(5, sum by (namespace, pod) (container_memory_working_set_bytes{container!=""}))'
995418112  namespace=kube-system pod=kube-apiserver-gitops-control-plane
706035712  namespace=argocd pod=argocd-application-controller-0
579960832  namespace=monitoring pod=monitoring-grafana-64595f4b85-9zr77
512368640  namespace=monitoring pod=prometheus-monitoring-kube-prometheus-prometheus-0
111087616  namespace=kube-system pod=etcd-gitops-control-plane

container!="" leaves out a duplicate the kubelet reports for each pod as a whole, which would count everything twice. sort_desc puts the biggest first; topk(5, ...) keeps only the top five. The monitoring stack uses 1.3 GB, the Argo CD stack 1.0 GB, and the bookshop, the thing being monitored, 29 MB. That's normal for a small cluster, and worth knowing when someone asks why monitoring costs what it does. The same for CPU:

$ q 'sort_desc(sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=""}[5m])))'
0.12429724362997366  namespace=kube-system
0.047532398055475836  namespace=monitoring
0.025105003495560243  namespace=argocd
...
0.00011410322078297808  namespace=bookshop

Comparing two metrics

kube-state-metrics reports how many pods each Deployment wants, and how many are available. Subtract:

$ q 'kube_deployment_spec_replicas - kube_deployment_status_replicas_available'
0  namespace=monitoring deployment=monitoring-grafana
0  namespace=argocd deployment=argocd-redis
0  namespace=kube-system deployment=coredns
...
0  namespace=bookshop deployment=bookshop

Fifteen Deployments, all 0 short. As a filter, it's a "pods missing" alert:

$ q 'kube_deployment_spec_replicas > kube_deployment_status_replicas_available'
(no results)

No results means nothing is wrong. Both sides matched because both metrics come from kube-state-metrics with the same labels. When two metrics' labels don't line up exactly, nothing matches, and you get no results at all: that's what on (...) is for, saying which labels to match on, and ignoring the rest:

$ q 'kube_deployment_spec_replicas - on (namespace, deployment) kube_deployment_status_replicas_available'
0  namespace=monitoring deployment=monitoring-grafana
...

The same answer here, but it keeps working when one side has an extra label the other doesn't. ignoring (...) is its opposite: match on everything except those.

Percentiles from a histogram

Lesson 2's histogram had buckets. histogram_quantile() turns them into "how long did the slowest 1% take?" The Kubernetes API server's own request times, leaving out long-lived connections, which would swamp everything:

$ q 'histogram_quantile(0.99, sum by (le) (rate(apiserver_request_duration_seconds_bucket{verb!~"WATCH|CONNECT"}[5m])))'
0.02417888198757764

$ q 'histogram_quantile(0.5, sum by (le) (rate(apiserver_request_duration_seconds_bucket{verb!~"WATCH|CONNECT"}[5m])))'
0.003083810888252149

Read from the inside out: the rate of each bucket over 5 minutes, added up across everything except le, the bucket boundary, then the quantile. 0.99 is the 99th percentile, p99: 99% of requests took under 24 milliseconds. 0.5 is the median, p50: half took under 3. Broken down by kind of request, keeping verb as well as le:

$ q 'histogram_quantile(0.99, sum by (le, verb) (rate(apiserver_request_duration_seconds_bucket{verb!~"WATCH|CONNECT"}[5m])))'
0.004973627684964201  verb=GET
0.07000000000000002  verb=LIST
NaN  verb=APPLY
0.02415  verb=POST
0.024619847328244276  verb=PUT
0.0248  verb=PATCH
NaN  verb=DELETE

LIST is the slowest, at 70 milliseconds: it returns many objects at once. NaN, "not a number", means there were no APPLY or DELETE requests in those 5 minutes, so there was nothing to take a percentile of. The average, for comparison, is the _sum rate divided by the _count rate:

$ q 'sum(rate(apiserver_request_duration_seconds_sum{verb!~"WATCH|CONNECT"}[5m])) / sum(rate(apiserver_request_duration_seconds_count{verb!~"WATCH|CONNECT"}[5m]))'
0.0031626854041810807

3.2 milliseconds, close to the median. Averages hide the slow ones: a page that's fast for 99 people and takes ten seconds for one has a fine average. That's why alerts and promises about speed use percentiles. Note that a quantile is an estimate: Prometheus only knows which bucket each request fell in, and assumes they're spread evenly inside it. And how many requests, by response code:

$ q 'sum by (code) (rate(apiserver_request_total[5m]))'
3.988888888888889  code=200
0.06296296296296296  code=201
0.003703703703703704  code=0
0  code=404
0  code=500
...

Four requests a second, nearly all 200, OK. Errors as a fraction of everything is the same pattern you'll write for the bookshop in Lesson 4.

Looking back, and ahead

$ q 'node_memory_MemAvailable_bytes offset 10m'
3057721344  instance=172.18.0.3:9100

$ q 'node_memory_MemAvailable_bytes - node_memory_MemAvailable_bytes offset 10m'
-91205632  instance=172.18.0.3:9100

$ q 'max_over_time(sum(rate(node_cpu_seconds_total{mode!="idle",instance="172.18.0.2:9100"}[5m]))[1h:1m])'
0.3112790030183509

• offset 10m: the same query, 10 minutes ago. Available memory went down by 91 MB since then.
• [1h:1m] after a whole query is a subquery: run it once a minute for the last hour, then max_over_time takes the largest. The busiest 5 minutes of the last hour used 0.31 of a CPU.

And predict_linear() draws a straight line through a gauge's recent values and asks where it will be later, which is how "this disk will be full in 4 hours" alerts are written:

$ q 'node_filesystem_avail_bytes{mountpoint="/"}'
(no results)

Nothing to predict from: on kind, node-exporter doesn't report a filesystem mounted at /. No results isn't an error, and it isn't zero: it means no series matched. Run count by (mountpoint) (node_filesystem_avail_bytes) to see which mounts it does report.

A graph is a query, asked many times

Everything so far is an instant query: one answer, now. A graph is a range query: the same question asked at regular steps over a period. Grafana does this for every line it draws:

curl -sG http://localhost:9090/api/v1/query_range \
  --data-urlencode 'query=1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))' \
  --data-urlencode "start=$(( $(date +%s) - 600 ))" --data-urlencode "end=$(date +%s)" \
  --data-urlencode "step=60" | jq -r '.data.result[0].values[] | "\(.[0] | floor | todate)  \(.[1])"'

start and end are times in seconds, the last 10 minutes; step=60 asks for one answer a minute. What came back is in Spot the problem, because it wasn't what it looked like.

Queries already in the stack

$ how many recording rules, and a few of their names
95
instance:node_cpu:rate:sum
instance:node_cpu:ratio
instance:node_cpu_utilisation:rate5m
node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate

$ q 'instance:node_cpu_utilisation:rate5m'
0.026192667516028778  instance=172.18.0.3:9100
0.025956559902813092  instance=172.18.0.2:9100

A recording rule is a query Prometheus runs every 30 seconds, saving the answer as a new metric. Its name says how it was made: instance (one per node), node_cpu_utilisation (what), rate5m (how). The query behind this one:

1 - avg without (cpu) (sum without (mode) (rate(node_cpu_seconds_total{job="node-exporter",mode=~"idle|iowait|steal"}[5m])))

Everything from this lesson: a rate, two aggregations with without, and 1 -. 2.6% busy, the same as your own query above. The chart came with 95 of these, so its dashboards and alerts don't work the same thing out from hundreds of series every time.

Mistakes that don't look like mistakes

$ q 'rate(node_memory_MemAvailable_bytes[5m])'
44504147.437037036  instance=172.18.0.3:9100

$ q 'deriv(node_memory_MemAvailable_bytes[5m])'
-29399.914037152826  instance=172.18.0.3:9100

rate() on a gauge: 44.5 million bytes a second. Memory isn't growing by 44 MB a second. rate() assumes a counter, so every time available memory went down, it treated that as the counter restarting from zero, and added the whole value back. The answer is a plausible-looking number, and meaningless. For a gauge, deriv() gives the real slope: memory going down by about 29 KB a second.

$ q 'rate(sum(node_cpu_seconds_total)[5m])'
error: invalid parameter "query": 1:33: parse error: ranges only allowed for vector selectors

$ q 'sum(rate(node_cpu_seconds_total[5m]))'
31.76462309607038

rate() first, then sum(). The window [5m] can only follow a metric's name, so the wrong order is at least a loud error. The right order works, though this one, summing every mode of every CPU on both nodes, adds up to "about 32 CPUs' worth of seconds per second", which is just the CPU count.

$ q 'node_memory_MemAvailable_bytes / node_memory_MemTotal'
(no results)

A misspelled name, MemTotal instead of MemTotal_bytes, isn't an error either. It's a series that doesn't exist, so nothing matches, and the division has nothing on one side. An alert written like this never fires.

Spot the problem

Work out what's wrong before you read the explanation.

The empty morning. The first time this lesson's queries ran, nearly every one with rate() came back (no results): CPU, percentiles, even the recording rule. Yet up said every target that should be up was up, and plain queries like node_memory_MemTotal_bytes answered fine. The range query, for the last 10 minutes, gave this:

2026-11-05T15:38:34Z  0.022194296576464878
2026-11-05T15:39:34Z  0.02471901439179769
2026-11-05T15:40:34Z  0.22136755176155565
2026-11-05T15:41:34Z  0.4167847064497855
2026-11-05T15:42:34Z  0.6125362767640293
2026-11-05T15:43:34Z  0.8083063211378861

Six points instead of eleven, and the CPU climbing from 2% to 81% in a straight line. What went wrong? The laptop had been asleep. It slept at 15:39 and woke at 15:47, and Docker's virtual machine, with the whole cluster, slept with it. Nothing was scraped for eight minutes. The queries ran seconds after waking, when the last 5 minutes held one sample or none, and rate() needs two. Plain queries still answered, because Prometheus returns a series' last sample if it's less than 5 minutes old. The straight-line climb wasn't the CPU: it was rate() running out of samples, its window holding fewer each minute, then nothing. Every target said up because each scrape that did happen worked. Ten minutes later, the same queries gave every answer in this lesson.

On a server, a laptop's sleep is a Prometheus restart, a network cut or a full disk. Three things to take from it. No results is a state, not an answer: check it before trusting it. A gap breaks rate() for one window after it, so a graph's odd shape right after a gap is the gap, not the system. And an alert can't fire on data that isn't there. Lesson 10's dead-man's switch is the answer to that.

Where you are now

You can ask Prometheus real questions: how busy, how full, how slow, what's missing, and what was it like an hour ago. You've also seen how a query can be wrong without saying so. Everything so far has been about Kubernetes and its machines. Lesson 4 gives the bookshop metrics of its own, so these same questions can be asked about its customers.

Check yourself

Key terms

• PromQL: Prometheus's query language.
• Label matcher: =, !=, =~ or !~, picking series by a label's value.
• Instant query: one answer, at one time.
• Range query: the same question at regular steps over a period, as a graph draws.
• Window: the period in brackets, such as [5m], that functions like rate() look back over.
• Aggregation: combining series, with sum, avg, max, count and others, by or without some labels.
• Percentile, quantile: the value a given share of things fall under. p99 is the 99th percentile.
• Subquery: a whole query run at steps over a period, written (QUERY)[1h:1m].
• Recording rule: a query Prometheus runs on a timer, saving the answer as a new metric.

Go deeper

• The Prometheus documentation: querying basics (opens in a new tab), operators (opens in a new tab) and functions (opens in a new tab), and histograms and summaries (opens in a new tab).
• Try: in Prometheus's web interface, run the CPU busy fraction from "Adding up" and switch to the Graph tab. Then try [1m] and [30s] in place of [5m]. What happens to the line, and why does [30s] show nothing?
• Try: write a query for "the share of API server requests that failed with a 5xx code over the last 5 minutes". Hint: code=~"5..", divided by everything.