Metrics and labels

By the end you'll know

What a target's /metrics page looks like, and how to read it
What a series is: a name and a set of labels
The four metric types, and how a counter and a gauge behave differently
How a histogram's buckets work
Which labels Prometheus adds, and where they come from
What cardinality is, and why it matters

Before you start

You need: Lesson 1 done, with the monitoring stack Synced and Healthy.
Nothing changes in this lesson. Every command only reads, from the cluster or from Prometheus.
Check where kubectl points: kubectl config current-context should say kind-gitops.
Where commands run: your own computer's terminal, in gitops-practice, with a Prometheus port-forward running.

The short version

A target serves text. node-exporter's page: 2,300 lines, each a name, some labels in braces, and a number.
A name isn't one number. node_cpu_seconds_total is 128 numbers on each node: 16 CPUs times 8 modes. Each name plus set of labels is a series.
Four types. A counter only goes up: CPU 0's idle time went from 189,617 to 189,646 seconds in 30 seconds. A gauge goes up and down. A histogram counts things into buckets. A summary is rarer.
Prometheus adds labels. A line with no labels on the exporter's page is stored with seven: job, instance, namespace, pod and more, from Kubernetes.
Series add up. 49,417 in this small cluster, and the biggest single name is 3,180 histogram buckets.

What a target serves

Lesson 1 said Prometheus scrapes targets. Here's what a scrape actually reads. node-exporter, one of the targets, has a Service in the monitoring namespace; a port-forward to it, and its page:

BASH

kubectl port-forward -n monitoring svc/monitoring-prometheus-node-exporter 9101:9100
curl -s http://localhost:9101/metrics | wc -l

OUTPUT

(The port-forward goes in its own terminal; 9101 on your side, so it doesn't clash with anything else.) 2,300 lines of plain text. That's the whole protocol: every 30 seconds, Prometheus asks for this page, and reads it. One metric from it:

BASH

curl -s http://localhost:9101/metrics | grep -A3 '^# HELP node_memory_MemAvailable_bytes'

OUTPUT

# HELP node_memory_MemAvailable_bytes Memory information field MemAvailable_bytes.
# TYPE node_memory_MemAvailable_bytes gauge
node_memory_MemAvailable_bytes 3.184214016e+09
# HELP node_memory_MemFree_bytes Memory information field MemFree_bytes.

Three kinds of line:

# HELP: what the metric means, in words.
# TYPE: what kind of metric it is. Here, a gauge.
The value: the metric's name, then its number. 3.184214016e+09 is scientific notation for 3,184,214,016: about 3.2 GB of memory available.

The name says what's measured, and the unit, at the end: _bytes. Prometheus names follow that pattern: _bytes, _seconds, and _total for counts.

One name, many series

Some metrics have more to say:

BASH

curl -s http://localhost:9101/metrics | grep -A4 '^# HELP node_cpu_seconds_total'

OUTPUT

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 189625.99
node_cpu_seconds_total{cpu="0",mode="iowait"} 30.04
node_cpu_seconds_total{cpu="0",mode="irq"} 0

The part in braces is labels: cpu="0", mode="idle". They split one name into many numbers, one for each combination: CPU 0's idle time, CPU 0's time waiting for the disk, and so on. How many lines, and how many of each mode?

OUTPUT

$ curl -s http://localhost:9101/metrics | grep '^node_cpu_seconds_total' | wc -l
     128

$ ... | sed -E 's/\{.*mode="([a-z]+)"\}.*/ mode=\1/' | sort | uniq -c
  16 node_cpu_seconds_total mode=idle
  16 node_cpu_seconds_total mode=iowait
  16 node_cpu_seconds_total mode=irq
  16 node_cpu_seconds_total mode=nice
  16 node_cpu_seconds_total mode=softirq
  16 node_cpu_seconds_total mode=steal
  16 node_cpu_seconds_total mode=system
  16 node_cpu_seconds_total mode=user

(sed cuts each line down to its mode, then sort | uniq -c counts them.) 16 CPUs, times 8 modes, is 128 lines from one name. Each name with its own set of labels is a series: Prometheus stores each one separately, as its own line of numbers over time. And CPU 0 has been idle for 189,626 seconds, about 52 hours. That counts from when the machine started, not the cluster: both kind nodes are containers on Docker's virtual machine, so the node-exporter on each reports that same machine, with the same 16 CPUs and the same memory. On real servers, each node would have its own.

The four types

The # TYPE lines, counted on three targets:

OUTPUT

--- node-exporter
 174 gauge
  87 counter
  49 untyped
   1 summary

--- kube-state-metrics
 179 gauge
   2 counter

--- prometheus
 129 counter
 109 gauge
  25 histogram
  15 summary

Counter: a number that only goes up, or goes back to zero when the program restarts. Requests served, seconds of CPU used, errors. Names end in _total.
Gauge: a number that goes up and down. Memory available, pods running, temperature.
Histogram: counts things into ranges, called buckets, such as how many requests took under 0.1 seconds, under 0.2, and so on. That's how Prometheus measures how slow something is.
Summary: like a histogram, but the program works out the answers itself. Rarer, and harder to combine across pods.
Untyped: the program didn't say. Prometheus treats it like a gauge.

kube-state-metrics is almost all gauges, because it reports how things are right now: pods wanted, pods available. Prometheus, measuring itself, has every kind.

A counter and a gauge, 30 seconds apart

The same two series, read from Prometheus twice, 30 seconds apart:

OUTPUT

first:
189617.62  ...cpu=0,instance=172.18.0.3:9100,mode=idle,...
3188768768  __name__=node_memory_MemAvailable_bytes,...instance=172.18.0.3:9100,...

30 seconds later:
189646.54  ...cpu=0,instance=172.18.0.3:9100,mode=idle,...
3280539648  __name__=node_memory_MemAvailable_bytes,...instance=172.18.0.3:9100,...

The counter went up by 28.92 seconds; the gauge went up by about 92 MB. Run it again and the gauge might go down; the counter never will. That's why their raw values mean different things. A gauge's value is the answer: 3.28 GB free. A counter's value on its own is almost useless, "189,646 seconds since some time two days ago". What matters is how fast it grows, which is what rate() works out:

BASH

q 'rate(node_cpu_seconds_total{mode="idle",cpu="0"}[5m])'

OUTPUT

0.9626367602723114  ...cpu=0,instance=172.18.0.3:9100,mode=idle,...
0.9624886110267312  ...cpu=0,instance=172.18.0.2:9100,mode=idle,...

q is the query function from Lesson 1. rate(...[5m]) means: over the last 5 minutes, how much did this counter grow per second? 0.96: CPU 0 spent 96% of its time idle. The 28.92 seconds in 30 agrees. Lesson 3 is all about questions like this one.

A histogram

Prometheus counts how long its own queries take. A histogram becomes one series per bucket, with a label, le, meaning "less than or equal to":

BASH

q 'prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query"}' \
  | jq -r '.data.result[] | "le=\(.metric.le)  \(.value[1])"' | sort -t= -k2 -g

OUTPUT

le=0.1  17
le=0.2  17
le=0.4  17
le=1.0  17
le=3.0  17
le=8.0  17
le=20.0  17
le=60.0  17
le=120.0  17
le=+Inf  17

$ q 'prometheus_http_request_duration_seconds_count{handler="/api/v1/query"}'
17  ...

17 queries so far, and every one took 0.1 seconds or less, so every bucket says 17. Buckets are cumulative: "under 0.2 seconds" includes everything under 0.1. +Inf, infinity, counts everything, and always equals _count. A slow query would show up as a jump: say 15 in 0.1 and 17 in 0.4 would mean two took between 0.1 and 0.4 seconds. Each bucket is a counter, so rate() works on them too, and Lesson 3 turns buckets into "how slow are the slowest 1% of requests".

The labels Prometheus adds

The same memory metric, on the exporter's page, then as Prometheus stored it:

OUTPUT

$ curl -s http://localhost:9101/metrics | grep '^node_memory_MemAvailable_bytes'
node_memory_MemAvailable_bytes 3.277492224e+09

$ q 'node_memory_MemAvailable_bytes'
3280539648  __name__=node_memory_MemAvailable_bytes,container=node-exporter,endpoint=http-metrics,
            instance=172.18.0.3:9100,job=node-exporter,namespace=monitoring,
            pod=monitoring-prometheus-node-exporter-smhnp,service=monitoring-prometheus-node-exporter

No labels on the page; seven in Prometheus. (And slightly different numbers: Prometheus's value is from its last scrape, a few seconds before.) Where did they come from? From the ServiceMonitor, and Kubernetes:

BASH

kubectl get servicemonitor -n monitoring monitoring-prometheus-node-exporter -o yaml | sed -n '/^spec:/,$p'

OUTPUT

spec:
  attachMetadata:
    node: false
  endpoints:
  - port: http-metrics
    scheme: http
  jobLabel: jobLabel
  selector:
    matchLabels:
      app.kubernetes.io/instance: monitoring
      app.kubernetes.io/name: prometheus-node-exporter

The ServiceMonitor says: find Services with these two labels, and scrape the port called http-metrics on every pod behind them. Prometheus asks Kubernetes for those pods, and gets back everything Kubernetes knows about each one. Those are the target's discovered labels, all starting with __:

OUTPUT

"discoveredLabels": {
  "__address__": "172.18.0.3:9100",
  "__meta_kubernetes_endpoint_node_name": "gitops-worker",
  "__meta_kubernetes_endpoint_port_name": "http-metrics",
  "__meta_kubernetes_namespace": "monitoring",
  "__meta_kubernetes_pod_name": "monitoring-prometheus-node-exporter-smhnp",
  "__meta_kubernetes_service_name": "monitoring-prometheus-node-exporter",
  "__metrics_path__": "/metrics",
  "__scheme__": "http"
},
"labels": {
  "container": "node-exporter",
  "endpoint": "http-metrics",
  "instance": "172.18.0.3:9100",
  "job": "node-exporter",
  "namespace": "monitoring",
  "pod": "monitoring-prometheus-node-exporter-smhnp",
  "service": "monitoring-prometheus-node-exporter"
}

(A few of the discovered labels, out of dozens.) Relabeling turns the first set into the second: __meta_kubernetes_namespace becomes namespace, the pod's name becomes pod, the address becomes instance. Labels starting with __ are dropped before anything is stored. job comes from jobLabel: jobLabel: use the value of the Service's own label called jobLabel, which is node-exporter. The operator writes all these rules for you, from the ServiceMonitor. Every series from that pod gets the same seven labels, which is how a query can ask "everything from the monitoring namespace" or "everything from this pod".

kube-state-metrics: Kubernetes as numbers

kube-state-metrics turns Kubernetes objects into series. The bookshop's Deployment, on its page:

OUTPUT

kube_deployment_status_replicas_available{namespace="bookshop",deployment="bookshop"} 2
kube_deployment_spec_replicas{namespace="bookshop",deployment="bookshop"} 2

Wanted 2, available 2. Together they're a question an alert can ask: is available less than wanted? And as Prometheus stored the first:

OUTPUT

2  __name__=kube_deployment_status_replicas_available,container=kube-state-metrics,deployment=bookshop,
   endpoint=http,instance=10.244.1.15:8080,job=kube-state-metrics,namespace=bookshop,
   pod=monitoring-kube-state-metrics-7994f48c57-fw69v,service=monitoring-kube-state-metrics

Look closely at two labels. namespace=bookshop: the metric's own label, kept, even though the target is in monitoring. The chart sets kube-state-metrics' ServiceMonitor to honor labels: when the page and Prometheus both have a label, the page's wins. And pod=monitoring-kube-state-metrics-...: that's not a bookshop pod. More on that in Spot the problem.

How many series

Every series costs memory, and Prometheus keeps the recent ones in memory, in what it calls the head. How many?

OUTPUT

$ q 'prometheus_tsdb_head_series'
49417  ...

$ q 'topk(8, count by (__name__) ({__name__=~".+"}))'
3180  __name__=apiserver_request_duration_seconds_bucket
3040  __name__=apiserver_request_body_size_bytes_bucket
2210  __name__=etcd_request_duration_seconds_bucket
1904  __name__=apiserver_request_sli_duration_seconds_bucket
1464  __name__=apiserver_response_sizes_bucket
1422  __name__=apiserver_watch_list_duration_seconds_bucket
966  __name__=apiserver_watch_cache_read_wait_seconds_bucket
654  __name__=kubernetes_feature_enabled

49,417 series, in a cluster running a bookshop and a guestbook. topk(8, ...) lists the eight metric names with the most series, and seven of them are histogram buckets, from the Kubernetes API server. One histogram is one series per bucket, for every combination of its other labels: every kind of resource, every kind of request, every response code. 3,180 series, from one name. Prometheus also reports which labels have the most different values:

BASH

curl -s http://localhost:9090/api/v1/status/tsdb | jq -r '.data.labelValueCountByLabelName[:8][] | "\(.value)  \(.name)"'

OUTPUT

2045  __name__
788  name
299  le
208  resource
120  method
95  id
74  handler
63  uid

2,045 different metric names; 299 different bucket boundaries (le); 63 different uids, one for each object kube-state-metrics reports on. The number of series a metric makes is its cardinality, and it multiplies: a histogram with 10 buckets, a label with 20 values and another with 5 is 1,000 series. That's fine. The trouble starts when a label's values never stop growing: a user ID, an email address, a full web address with a query in it. Every new value is a new series, forever, and a Prometheus that ran fine for months runs out of memory. Labels are for things with a small, fixed set of values. Lesson 4 keeps to that, for the bookshop.

A series, over time

How often does Prometheus scrape, and what does a series actually hold?

OUTPUT

$ curl -s http://localhost:9090/api/v1/status/config | jq -r .data.yaml | grep -m2 -e scrape_interval -e evaluation_interval
  scrape_interval: 30s
  evaluation_interval: 30s

$ q 'count_over_time(up{job="node-exporter"}[5m])'
10  ...instance=172.18.0.3:9100,job=node-exporter,...
10  ...instance=172.18.0.2:9100,job=node-exporter,...

$ q 'up{job="node-exporter"}[2m]' | jq -r '.data.result[0].values[] | "\(.[0] | floor | todate)  \(.[1])"'
2026-11-05T14:47:39Z  1
2026-11-05T14:48:09Z  1
2026-11-05T14:48:39Z  1
2026-11-05T14:49:09Z  1

A scrape every 30 seconds, so 10 samples in 5 minutes, and 4 in 2. Each sample is a time and a number, here 1, for up. That's all a series is: a name, its labels, and a line of samples. [2m] after a name asks for the samples themselves, a range, instead of only the latest one. And evaluation_interval: every 30 seconds, Prometheus also checks its 155 alert rules. That, with for: 600s, is why Lesson 1's alerts took 10 minutes to fire.

In real systems

Cardinality is the usual way Prometheus falls over. Teams watch prometheus_tsdb_head_series, and alert when it jumps. A release that adds a label with a user ID in it shows up there first.
Drop what nobody uses. Those 3,180 API server buckets cost memory whether anyone looks at them or not. A ServiceMonitor can drop metrics before they're stored, with metricRelabelings.
Scrape interval is a trade-off. 30 seconds is the default here. Shorter catches quicker spikes and costs more series data; longer misses short problems. Most teams stay between 15 seconds and a minute.
Name things the Prometheus way. Base units (_seconds, not milliseconds; _bytes, not megabytes), _total on counters. Lesson 4 follows this for the bookshop.

Spot the problem

Work out what's wrong before you read the explanation.

The pod that wasn't. An alert says the bookshop has one pod available instead of two. The engineer on call queries the bookshop's kube_deployment_status_replicas_available, sees the series' labels, and copies the pod name from them: pod=monitoring-kube-state-metrics-7994f48c57-fw69v. They run kubectl describe pod monitoring-kube-state-metrics-7994f48c57-fw69v -n bookshop, and Kubernetes says there's no such pod. What went wrong? That pod label is about the scrape, not the bookshop. Prometheus adds pod to every series from a target: the pod it scraped. For anything from kube-state-metrics, that's kube-state-metrics itself, in monitoring. A Deployment metric is about a Deployment, so it has no pod of its own, and the scraped pod's name is the only pod label it gets. Metrics that are about a pod, like the restart count, carry their own pod label, and honor labels keeps it: Lesson 1's restart counts said bookshop-74d9f67bc8-4kf4c. The fix is to find the pods another way: kubectl get pods -n bookshop, or the restart and readiness metrics, which name them. Before trusting a label, know who set it: the program that made the metric, or Prometheus, describing where it scraped.

Keep this

To	Run
See what a target serves	`kubectl port-forward -n NS svc/SERVICE LOCAL:PORT`, then `curl -s localhost:LOCAL/metrics`
What a metric means, and its type	`grep -A3 '^# HELP NAME'` on that page
Count types on a page	`awk '/^# TYPE/{print $4}' \| sort \| uniq -c`
How fast a counter grows	`rate(NAME[5m])`
Count series for a name	`count(NAME)`, or `count by (LABEL) (NAME)`
A histogram's buckets	`NAME_bucket`, with `le`; `NAME_count` and `NAME_sum`
A target's discovered labels	`curl -s localhost:9090/api/v1/targets \| jq '.data.activeTargets[0].discoveredLabels'`
How a ServiceMonitor picks targets	`kubectl get servicemonitor -n monitoring NAME -o yaml`
Total series in memory	`prometheus_tsdb_head_series`
The names with the most series	`topk(8, count by (name) ({name=~".+"}))`
The labels with the most values	`curl -s localhost:9090/api/v1/status/tsdb`
A series' raw samples	`NAME[2m]`

Where you are now

You can read a target's page, tell a counter from a gauge, read a histogram's buckets, and say where every label on a series came from. You've also seen the cost of all of it: 49,417 series, mostly buckets nobody looks at. Lesson 3 starts asking real questions of them, with PromQL.

Check yourself

What is a series?
node_cpu_seconds_total went from 189,617 to 189,646. What kind of metric is it?
Every bucket of the query histogram said 17. What did that mean?
The exporter's line had no labels. Where did job, pod and namespace come from?
Why is a user ID a bad label?
With a 30-second scrape interval, how many samples does [5m] hold?

Key terms

Sample: one number at one time.
Series: a metric name with one set of labels, and its samples over time.
Label: a name and value that tells series of the same metric apart, such as mode="idle".
Counter: a number that only goes up, until its program restarts.
Gauge: a number that goes up and down.
Histogram: counts values into cumulative buckets, labelled le.
Relabeling: Prometheus's rules for turning what it discovered about a target into labels.
Honor labels: keeping the page's own label when Prometheus would add one with the same name.
Cardinality: how many series a metric makes, from every combination of its labels.
Head: the recent series Prometheus keeps in memory.

Go deeper

The Prometheus documentation: data model (opens in a new tab), metric types (opens in a new tab) and naming (opens in a new tab).
Try: in Prometheus's web interface, open Status, TSDB Status. Which metric names and labels top its lists, and do they match the ones above?
Try: query the bookshop pods' restart counts, the query from Lesson 1's "What it can see about the bookshop". Which pod label do they have, and why is it different from the Deployment metric's?