The bookshop's own metrics

Where a program's metrics come from

Lesson 1 ended with 125 metrics about the bookshop and not one from it. Kubernetes can describe a pod from the outside: running, restarting, using memory. Only the program inside can say what it's doing: how many requests, how many failed, how slow.

There are two ways to get those numbers:

• The program serves them itself, on a /metrics page, because its authors added that. Prometheus, Grafana, Argo CD and most tools written in the last few years do. Your own apps can, with a Prometheus client library for their language.
• An exporter translates. The program reports its status some other way, and a small separate program, an exporter, reads that and serves it as metrics. node-exporter does this for a machine. For nginx, it's the NGINX Prometheus Exporter, which reads nginx's built-in status page.

The open-source nginx doesn't serve Prometheus metrics itself, so the bookshop gets an exporter.

The status page

nginx has a module called stub_status that serves a short page of counts. It's turned on with a few lines of nginx configuration, a new template in the chart, templates/nginx-conf.yaml:

{{- if .Values.metrics.enabled }}
# nginx's built-in status page, for the exporter next to it (Monitoring track, Lesson 4).
# Its own port, 8080, and only from inside the pod: the Service never sends traffic here.
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-nginx-conf
  labels:
    {{- include "bookshop.labels" . | nindent 4 }}
data:
  status.conf: |
    server {
      listen 8080;
      location /stub_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
      }
    }
{{- end }}

A second nginx server, on port 8080, with one address, /stub_status. allow 127.0.0.1; deny all; means only requests from inside the pod itself, from localhost, get an answer. The status page isn't secret, but it's nobody else's business, and it's not for customers. The whole thing is behind metrics.enabled, a new value, so it can be switched off.

The exporter, beside nginx

A pod can have more than one container. They share the pod's network, so they can reach each other on localhost, and they start and stop together. A helper container like this, next to the main one, is called a sidecar. In templates/deployment.yaml, three additions. The configuration file, mounted into nginx's configuration folder:

          volumeMounts:
            - name: page
              mountPath: /usr/share/nginx/html
            {{- if .Values.metrics.enabled }}
            - name: nginx-conf
              mountPath: /etc/nginx/conf.d/status.conf
              subPath: status.conf
            {{- end }}

subPath mounts one file from the ConfigMap, rather than the whole ConfigMap as a folder, which would hide nginx's own configuration files in that folder. Then the exporter, as a second container:

        {{- if .Values.metrics.enabled }}
        # Reads nginx's status page and serves it as Prometheus metrics (Monitoring track, Lesson 4).
        - name: exporter
          image: "{{ .Values.metrics.image.repository }}:{{ .Values.metrics.image.tag }}"
          args:
            - --nginx.scrape-uri=http://localhost:8080/stub_status
          ports:
            - name: metrics
              containerPort: 9113
          resources:
            {{- toYaml .Values.metrics.resources | nindent 12 }}
        {{- end }}

It reads http://localhost:8080/stub_status, nginx in the same pod, and serves metrics on port 9113, named metrics. And a checksum, like the page's, so a change to the status configuration rolls the pods:

        {{- if .Values.metrics.enabled }}
        checksum/nginx-conf: {{ include (print $.Template.BasePath "/nginx-conf.yaml") . | sha256sum }}
        {{- end }}

The Service gets a second port, so Prometheus can find the exporter by name:

  ports:
    - name: http
      port: 80
      targetPort: 80
    {{- if .Values.metrics.enabled }}
    - name: metrics
      port: 9113
      targetPort: metrics
    {{- end }}

And the new values, in values.yaml, with the exporter's version pinned:

metrics:
  enabled: true     # nginx's status page, an exporter next to nginx, and a ServiceMonitor
  image:
    repository: nginx/nginx-prometheus-exporter
    tag: "1.5.3"
  resources:
    requests:
      cpu: 5m
      memory: 16Mi
    limits:
      memory: 32Mi
  serviceMonitor:
    labels: {}      # extra labels for the ServiceMonitor

1.5.3 was the newest release on Docker Hub. Check it exists before committing, as in Argo CD Lesson 7: docker buildx imagetools inspect nginx/nginx-prometheus-exporter:1.5.3.

The ServiceMonitor

Last, templates/servicemonitor.yaml: the instruction to Prometheus, from Lesson 1's list of new resource kinds.

{{- if .Values.metrics.enabled }}
# Tells Prometheus to scrape the exporter on every bookshop pod (Monitoring track, Lesson 4).
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ .Release.Name }}
  labels:
    {{- include "bookshop.labels" . | nindent 4 }}
    {{- with .Values.metrics.serviceMonitor.labels }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: {{ .Release.Name }}
  endpoints:
    - port: metrics
      interval: 30s
{{- end }}

Find the Service with this release's label, and scrape its port called metrics every 30 seconds, on every pod behind it. {{- with ... }} adds any extra labels from the values; there are none yet. Checked, then merged:

$ helm lint charts/bookshop
==> Linting charts/bookshop
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

$ helm template ... -f values-production.yaml | grep -c 'kind:'
6

merged: b2e5b74 Bookshop: an nginx exporter and a ServiceMonitor (#42)
deployment "bookshop" successfully rolled out

$ kubectl get pods -n bookshop -l app=bookshop
NAME                        READY   STATUS    RESTARTS   AGE
bookshop-65d8d9ff66-2krfn   2/2     Running   0          17s
bookshop-65d8d9ff66-6dwvg   2/2     Running   0          15s

Six resources for production now: the page, the status configuration, the Service, the Deployment, the sealed password and the ServiceMonitor. And 2/2: two containers in each pod, nginx and the exporter, both ready. Staging rolled too, because metrics.enabled defaults to true for every environment.

What the exporter serves

A port-forward straight to one pod. It arrives inside the pod as localhost, so the status page answers:

kubectl port-forward -n bookshop pod/POD 18080:8080 19113:9113
curl -s http://localhost:18080/stub_status

Active connections: 1 
server accepts handled requests
 3 3 3 
Reading: 0 Writing: 1 Waiting: 0 

nginx's own words: one connection open right now (this one), three accepted since it started, three handled, three requests. And what the exporter made of it a moment later:

$ curl -s http://localhost:19113/metrics | grep '^nginx_'
nginx_connections_accepted 4
nginx_connections_active 2
nginx_connections_handled 4
nginx_connections_reading 0
nginx_connections_waiting 1
nginx_connections_writing 1
nginx_exporter_build_info{branch="main",goarch="arm64",...} 1
nginx_http_requests_total 4
nginx_up 1

The same numbers, as metrics, named the Prometheus way: _total on the request counter, the rest gauges. nginx_up 1 means the exporter could read the status page; if nginx stopped answering, it would turn 0 while the exporter itself stayed up. (grep -c '^# TYPE' counted 49 metrics in all: the rest are about the exporter itself, as a Go program.)

And is the status page really private? From another pod, the curl probe pod from the Argo CD track:

$ kubectl exec -n bookshop probe -- curl -s -o /dev/null -w '%{http_code}\n' http://POD-IP:8080/stub_status
403

$ kubectl exec -n bookshop probe -- curl -s -o /dev/null -w '%{http_code}\n' http://bookshop.bookshop/
200

$ kubectl exec -n bookshop probe -- curl -s http://bookshop.bookshop:9113/metrics | grep '^nginx_up'
nginx_up 1

403, Forbidden, from outside the pod; the shop, 200. The metrics port answers anyone in the cluster, which is what Prometheus needs.

And Prometheus?

A minute later, Prometheus should have three new targets: two in production, one in staging.

$ kubectl get servicemonitor -A
NAMESPACE          NAME                                                 AGE
bookshop-staging   bookshop                                             72s
bookshop           bookshop                                             83s
monitoring         monitoring-grafana                                   14h
...

$ bookshop_targets
(no bookshop targets)

(bookshop_targets is a short function that lists any target in either bookshop namespace.) The ServiceMonitors exist, and Prometheus has no bookshop targets. No error, nowhere. The reason is in the Prometheus resource, the one the operator reads to run Prometheus:

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'
{"matchLabels":{"release":"monitoring"}}

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorNamespaceSelector}'
{}

serviceMonitorSelector: only ServiceMonitors labelled release: monitoring. Every namespace is allowed ({} means no limit), but only ServiceMonitors with that label. The chart's own ones all have it:

$ kubectl get servicemonitor -n monitoring monitoring-prometheus-node-exporter -o jsonpath='{.metadata.labels}' | jq .
{
  "app.kubernetes.io/component": "metrics",
  "app.kubernetes.io/instance": "monitoring",
  ...
  "release": "monitoring"
}

The chart does this on purpose, so that if a cluster runs two Prometheus stacks, each picks up only its own. release is its release name, monitoring, from the Application in Lesson 1. The bookshop's ServiceMonitor needs it too. That's what metrics.serviceMonitor.labels is for:

   serviceMonitor:
-    labels: {}      # extra labels for the ServiceMonitor
+    labels:           # extra labels for the ServiceMonitor
+      release: monitoring   # kube-prometheus-stack's Prometheus only picks up ServiceMonitors with this

merged: 2486822 Bookshop: the ServiceMonitor label Prometheus selects on (#43)

all bookshop targets up after about 60s

$ bookshop_targets
up  job=bookshop namespace=bookshop-staging pod=bookshop-6869cf94c9-z5jcl
up  job=bookshop namespace=bookshop pod=bookshop-65d8d9ff66-2krfn
up  job=bookshop namespace=bookshop pod=bookshop-65d8d9ff66-6dwvg

Three targets, all up, 60 seconds later. The operator noticed the label, rewrote Prometheus's settings, and Prometheus reloaded them. job=bookshop for both environments: the job comes from the Service's name, which is bookshop in each. The namespace tells them apart.

The label lives in the chart's values, not in the template, because it's about this cluster's Prometheus, not about the bookshop. A cluster with a differently configured Prometheus would set a different label, or none.

Nobody visiting

Ninety seconds later, before anyone visits:

$ q 'nginx_http_requests_total{namespace="bookshop"}'
29  pod=bookshop-65d8d9ff66-6dwvg
34  pod=bookshop-65d8d9ff66-2krfn

$ q 'sum by (namespace) (rate(nginx_http_requests_total[2m]))'
0.1333333333333333  namespace=bookshop-staging
0.26665777837033083  namespace=bookshop

q is Lesson 3's query function. Requests, with nobody visiting: about 0.13 a second for each pod. Two things ask every pod for a page all the time:

• The readiness check, from Argo CD Lesson 3. The kubelet asks for / every 10 seconds: 0.1 a second.
• The exporter. Each time Prometheus scrapes it, every 30 seconds, it reads the status page, and that's a request too: 0.033 a second.

0.1 plus 0.033 is 0.133, exactly what staging's one pod shows, and production's two pods are twice that. Worth knowing before trusting the counter as "customers": a quiet shop isn't zero. It's whatever the machinery does on its own.

Some customers

The probe pod, asking for the shop's page twice a second, for three minutes:

352 requests sent

$ q 'sum by (namespace) (rate(nginx_http_requests_total[2m]))'
0.1333777925975325  namespace=bookshop-staging
2.1897274817142893  namespace=bookshop

$ q 'sum by (pod) (rate(nginx_http_requests_total{namespace="bookshop"}[2m]))'
1.1671075739723897  pod=bookshop-65d8d9ff66-6dwvg
1.0226199077418996  pod=bookshop-65d8d9ff66-2krfn

$ q 'sum(increase(nginx_http_requests_total{namespace="bookshop"}[3m]))'
400.8855222447456

Production went up to 2.19 a second; staging stayed at 0.13. The probe aimed at about 2 a second, and each request plus its half-second pause took a little longer, so 352 in 180 seconds: 1.96 a second. Plus the 0.27 baseline, that's about 2.22, close to the 2.19 measured. The Service shared them about evenly between the two pods, 1.17 and 1.02. And increase over the 3 minutes: 401 requests, which is the 352, plus about 48 from the machinery. The numbers agree with what was sent, which is the test any new metric should pass before you build on it.

What it can't tell you

The full list of what the status page provides:

# HELP nginx_connections_accepted Accepted client connections
# HELP nginx_connections_active Active client connections
# HELP nginx_connections_handled Handled client connections
# HELP nginx_connections_reading Connections where NGINX is reading the request header
# HELP nginx_connections_waiting Idle client connections
# HELP nginx_connections_writing Connections where NGINX is writing the response back to the client
# HELP nginx_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, gove...
# HELP nginx_http_requests_total Total http requests
# HELP nginx_up Status of the last metric scrape

Connections, and one count of requests, with no labels. Then a request for a page that doesn't exist:

$ kubectl exec -n bookshop probe -- curl -s -o /dev/null -w '%{http_code}\n' http://bookshop.bookshop/no-such-page
404

404, Not Found, and it went into nginx_http_requests_total exactly like a good request. There's no code label, so a shop serving nothing but errors would look as busy as a shop selling books. And no histogram, so nothing about how slow. Lesson 3's questions, "what share failed?" and "how slow is p99?", can't be asked of this metric at all.

That's the honest limit of nginx's free status page, and of most exporters: you get what the program reports, and no more. The ways past it:

• Instrument the app itself. A real shop is code, not nginx serving one file, and its code would count requests by status code and time them, with a client library.
• Read the access log. nginx writes a line for every request, with its status code and how long it took. Lesson 9 brings logs in, with Loki.
• Check from the outside. Ask for the page the way a customer does, and record whether it worked and how long it took. That's Lesson 5.

Spot the problem

Work out what's wrong before you read the explanation.

The target that never came. A team adds a ServiceMonitor for their new service, copied from a working one in another team's repository. Argo CD shows it Synced and Healthy. kubectl get servicemonitor lists it. Prometheus's Targets page doesn't, and there are no errors anywhere: not in Argo CD, not in Prometheus's logs, not in the operator's. After an hour, someone suggests the exporter is broken, and they spend the afternoon on that. What went wrong? The ServiceMonitor didn't match Prometheus's serviceMonitorSelector. The one they copied had a release: label for a different Prometheus, or the label was dropped in the copy. A ServiceMonitor that no Prometheus selects is valid YAML, a perfectly healthy resource, and completely ignored, so nothing reports a problem. The first thing to check, before the exporter, is the selector, next to the ServiceMonitor's own labels:

kubectl get prometheus -A -o jsonpath='{.items[*].spec.serviceMonitorSelector}'
kubectl get servicemonitor -n NAMESPACE NAME -o jsonpath='{.metadata.labels}'

It's the same lesson as Lesson 3's empty answers: nothing happening isn't the same as nothing wrong.

Where you are now

The bookshop reports on itself: three targets, a request rate that agrees with what was sent, and a baseline you can explain. You also know where it stops: no failures, no slowness. Lesson 5 checks the shop the way a customer would, from outside, so "is the page actually working, and how fast" gets an answer too.

Check yourself

Key terms

• Exporter: a program that reads another program's status and serves it as Prometheus metrics.
• Sidecar: a helper container in the same pod as the main one, sharing its network.
• stub_status: nginx's built-in status page: connections and a count of requests.
• subPath: mounting one file from a volume, instead of the whole volume as a folder.
• ServiceMonitor: tells Prometheus to scrape a named port on the pods behind matching Services.
• serviceMonitorSelector: the labels a Prometheus requires before it uses a ServiceMonitor.
• Instrumenting: adding metrics to an app's own code, with a client library.
• Baseline: what a metric reads when nothing in particular is happening.

Go deeper

• The NGINX Prometheus Exporter (opens in a new tab), nginx's stub_status module (opens in a new tab), and the Prometheus Operator's ServiceMonitor documentation (opens in a new tab).
• Try: in Grafana, use Explore, choose Prometheus, and graph sum by (namespace) (rate(nginx_http_requests_total[2m])). Send some traffic from the probe pod and watch it rise.
• Try: set metrics.enabled: false in values-staging.yaml, through a pull request. What happens to staging's pods, Service and target? Put it back afterwards.