Checking from the outside

By the end you'll know

The difference between checking from the inside and from the outside
What the blackbox exporter does, and what a module is
How to ask it by hand, and read what it says
How a Probe resource makes Prometheus run a check on a timer
How a content check catches a release that's healthy and wrong
Why up stays 1 when a check fails, and what to alert on instead

Before you start

You need: Lesson 4 done, and the Argo CD port-forward running, with argocd login done.
This lesson changes things: four pull requests. A new Application, two Probes, and a bad release on staging that you'll revert.
Check where kubectl points: kubectl config current-context should say kind-gitops.
Where commands run: your own computer's terminal, in gitops-practice.

The short version

The blackbox exporter. Give it an address and a kind of check, a module, and it asks for the page and reports back: success, status code, size, and the time each step took.
By hand. The shop: 200, in 3.4 milliseconds. A missing page: 404, failed. An address that doesn't exist: failed at the name lookup.
On a timer, from Git. Two Probes in apps/monitoring/: is each environment answering, and does each page say the banner, word for word.
Healthy and wrong. The typo banner on staging. Argo CD Synced Healthy, the pod available, nginx answering 200. The banner check failed 40 seconds after the release, and passed again 20 seconds after the revert.
The catch. up stayed 1 the whole time. up is about the check running, not the check passing: alert on probe_success.

Inside and outside

Everything so far has checked the bookshop from the inside. Kubernetes asks: is the pod running, is it ready? nginx says how many requests it has served. Each is true, and none of them is what a customer does. A customer asks for the page and looks at it.

Two names for the two views:

White-box monitoring: from the inside, with what the system reports about itself. Metrics like Lesson 4's. It tells you why something is wrong.
Black-box monitoring: from the outside, as a user, knowing nothing about how the system works. It tells you that something is wrong, whatever the cause.

Real setups use both. The outside check is the one that matches what customers see, and the one that catches failures nobody thought to measure from the inside.

The blackbox exporter

The blackbox exporter is an exporter with no program of its own to report on. You give it an address and a module, a description of what to check, and it does the check right then and serves the results as metrics. The chart comes with modules like http_2xx: ask for the address over HTTP, and pass if the answer is a 2xx status code, meaning success. It lives in the same chart repository as everything else:

OUTPUT

$ helm search repo prometheus-community/prometheus-blackbox-exporter
NAME                                             	CHART VERSION	APP VERSION	DESCRIPTION
prometheus-community/prometheus-blackbox-exporter	11.20.2      	v0.29.1    	Prometheus Blackbox Exporter

One new Application in argocd/, with one module of our own added:

YAML

# The blackbox exporter (Monitoring track, Lesson 5): checks addresses from the outside,
# the way a visitor would, and reports whether they worked and how long they took.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: blackbox-exporter
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    chart: prometheus-blackbox-exporter
    targetRevision: 11.20.2
    helm:
      releaseName: blackbox-exporter
      valuesObject:
        fullnameOverride: blackbox-exporter
        config:
          modules:
            # Added to the chart's own modules, such as http_2xx: the page must say
            # the banner, word for word, as well as answer.
            http_bookshop_banner:
              prober: http
              timeout: 5s
              http:
                preferred_ip_protocol: ip4
                fail_if_body_not_matches_regexp:
                  - "Free delivery on every order this week"
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

http_bookshop_banner checks over HTTP, like http_2xx, and also fails unless the page contains that text, the banner, exactly. That's a content check: not "did it answer?" but "did it answer with the right thing?" preferred_ip_protocol: ip4 says to use IPv4 addresses first, which is all a kind cluster has. Helm merges values, so the chart's own modules stay. fullnameOverride gives it a short name, blackbox-exporter, for its Service.

OUTPUT

merged: b002198 Monitoring: the blackbox exporter, installed by Argo CD (#44)
deployment "blackbox-exporter" successfully rolled out
Synced Healthy 11.20.2

NAME                                    READY   STATUS    RESTARTS   AGE
pod/blackbox-exporter-5b6bbd5c6-lzrq8   1/1     Running   0          9s

NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/blackbox-exporter   ClusterIP   10.96.240.81   <none>        9115/TCP   9s

Asking it by hand

A port-forward to its port, 9115, and a check. The address to ask about, and the module, go in the web address:

BASH

kubectl port-forward -n monitoring svc/blackbox-exporter 9115:9115
curl -s 'http://localhost:9115/probe?module=http_2xx&target=http://bookshop.bookshop.svc/' | grep -v '^#'

OUTPUT

probe_duration_seconds 0.003439375
probe_http_content_length 7724
probe_http_status_code 200
probe_success 1

(The full answer has more; these are the main ones.) http://bookshop.bookshop.svc/ is the bookshop Service's address inside the cluster: SERVICE.NAMESPACE.svc. Success, 200, 7,724 bytes, in 3.4 milliseconds. And the time, step by step:

OUTPUT

probe_http_duration_seconds{phase="connect"} 0.000128625
probe_http_duration_seconds{phase="processing"} 0.000209583
probe_http_duration_seconds{phase="resolve"} 0.001240708
probe_http_duration_seconds{phase="tls"} 0
probe_http_duration_seconds{phase="transfer"} 5.275e-05

resolve: looking up the name, bookshop.bookshop.svc, to get an address. 1.2 milliseconds: the slowest step, here.
connect: opening the connection.
tls: setting up encryption. 0: this is plain HTTP, inside the cluster.
processing: from asking to the first byte of the answer. nginx's own time.
transfer: receiving the rest of the page.

When a site is slow, this says where: the name lookup, the network, the encryption, or the app. Now a page that doesn't exist:

OUTPUT

$ curl -s '...?module=http_2xx&target=http://bookshop.bookshop.svc/no-such-page' | grep -e '^probe_success' -e '^probe_http_status_code'
probe_http_status_code 404
probe_success 0

404, and failed, because http_2xx wants a 2xx. Lesson 4's nginx metrics counted a 404 like a sale; this check doesn't. Add &debug=true, and it says why:

OUTPUT

level=ERROR source=http.go:601 msg="Invalid HTTP response status code, wanted 2xx" module=http_2xx target=http://bookshop.bookshop.svc/no-such-page status_code=404
level=ERROR source=handler.go:140 msg="Probe failed" module=http_2xx target=http://bookshop.bookshop.svc/no-such-page duration_seconds=0.001582333

The banner check, and an address that doesn't exist at all:

OUTPUT

$ curl -s '...?module=http_bookshop_banner&target=http://bookshop.bookshop.svc/' | grep '^probe_success'
probe_success 1

$ curl -s '...?module=http_2xx&target=http://no-such-shop.bookshop.svc/' | grep -e '^probe_success' -e '^probe_dns_lookup_time_seconds'
probe_dns_lookup_time_seconds 0.030564083
probe_success 0

The banner is there, so it passes. And the made-up address fails before it gets as far as asking: there's no such name to look up. A shop whose name stopped resolving would fail this way, and every inside check would still say the pods were fine.

Probes: on a timer, from Git

Asking by hand is a test. To ask every 30 seconds, and keep the answers, Prometheus does the asking: it scrapes the blackbox exporter's /probe address with each target and module, the same as curl did. A Probe, another of Lesson 1's resource kinds, describes that, and the operator turns it into Prometheus's settings. Lesson 4 taught the first thing to check:

OUTPUT

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.probeSelector}'
{"matchLabels":{"release":"monitoring"}}

The same rule as for ServiceMonitors: only Probes labelled release: monitoring. So the label goes in from the start. Two Probes, in a new file, apps/monitoring/bookshop-probes.yaml, where the monitoring-config Application from Lesson 1 applies it:

YAML

# Checks the bookshop from the outside (Monitoring track, Lesson 5), through the blackbox exporter.
# The release label is what this cluster's Prometheus looks for, as with ServiceMonitors.
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: bookshop-up
  labels:
    release: monitoring
spec:
  jobName: bookshop-up
  interval: 30s
  module: http_2xx
  prober:
    url: blackbox-exporter.monitoring.svc:9115
  targets:
    staticConfig:
      static:
        - http://bookshop.bookshop.svc/
        - http://bookshop.bookshop-staging.svc/
        - http://bookshop.bookshop.svc/no-such-page
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: bookshop-banner
  labels:
    release: monitoring
spec:
  jobName: bookshop-banner
  interval: 30s
  module: http_bookshop_banner
  prober:
    url: blackbox-exporter.monitoring.svc:9115
  targets:
    staticConfig:
      static:
        - http://bookshop.bookshop.svc/
        - http://bookshop.bookshop-staging.svc/

prober: where the blackbox exporter is.
module: which check.
targets: the addresses to check. bookshop-up checks both environments, and the missing page, on purpose, to see a failure. bookshop-banner checks that each page says the right thing.
jobName: the job label the results get.

OUTPUT

merged: 6decee9 Monitoring: probes for the bookshop, from the outside (#45)

$ kubectl get probes -n monitoring
NAME              AGE
bookshop-banner   6s
bookshop-up       6s

all 5 probe series in Prometheus after about 80s

Five checks, the three addresses and two pages, in Prometheus 80 seconds later.

What the probes say

OUTPUT

$ q 'probe_success'
1  job=bookshop-banner instance=http://bookshop.bookshop-staging.svc/
1  job=bookshop-up instance=http://bookshop.bookshop-staging.svc/
1  job=bookshop-up instance=http://bookshop.bookshop.svc/
1  job=bookshop-banner instance=http://bookshop.bookshop.svc/
0  job=bookshop-up instance=http://bookshop.bookshop.svc/no-such-page

$ q 'probe_http_status_code'
200  job=bookshop-banner instance=http://bookshop.bookshop-staging.svc/
200  job=bookshop-up instance=http://bookshop.bookshop-staging.svc/
200  job=bookshop-up instance=http://bookshop.bookshop.svc/
200  job=bookshop-banner instance=http://bookshop.bookshop.svc/
404  job=bookshop-up instance=http://bookshop.bookshop.svc/no-such-page

q is Lesson 3's query function. Four passes, and the one failure that's meant to fail. The address is the instance label. And the sizes:

OUTPUT

$ q 'probe_http_content_length{job="bookshop-up"}'
7754  instance=http://bookshop.bookshop-staging.svc/
7724  instance=http://bookshop.bookshop.svc/
153  instance=http://bookshop.bookshop.svc/no-such-page

Staging's page is 30 bytes bigger: its name, The Bookshop (staging), appears three times. nginx's "not found" page is 153 bytes. And one more:

OUTPUT

$ q 'up{job=~"bookshop-.*"}'
1  job=bookshop-banner instance=http://bookshop.bookshop-staging.svc/
1  job=bookshop-up instance=http://bookshop.bookshop-staging.svc/
1  job=bookshop-up instance=http://bookshop.bookshop.svc/
1  job=bookshop-banner instance=http://bookshop.bookshop.svc/
1  job=bookshop-up instance=http://bookshop.bookshop.svc/no-such-page

up is 1 for the missing page too. up says whether Prometheus could scrape the blackbox exporter, and it could: the exporter answered, with "the check failed". The check's result is probe_success. An alert on up for a probe would never fire, however broken the shop got. Alert on probe_success == 0.

Healthy and wrong, on staging

Argo CD Lesson 9 shipped a banner full of typos, and everything said Healthy. The same release, on staging, with the checks watching:

OUTPUT

-  banner: Free delivery on every order this week.
+  banner: Free delivery on evry order this weak.

merged: 75448f0 Staging: a new banner (#46)
deployment "bookshop" successfully rolled out
Synced Healthy 75448f047440560

the banner check failed after about 40s

Synced Healthy, and 40 seconds later, one check failed. Everything about staging, side by side:

OUTPUT

$ q 'probe_success{instance=~".*bookshop-staging.*"}'
0  job=bookshop-banner instance=http://bookshop.bookshop-staging.svc/
1  job=bookshop-up instance=http://bookshop.bookshop-staging.svc/

$ q 'probe_http_status_code{instance=~".*bookshop-staging.*"}'
200  job=bookshop-banner
200  job=bookshop-up

$ q 'probe_failed_due_to_regex{instance=~".*bookshop-staging.*"}'
1  job=bookshop-banner
0  job=bookshop-up

$ q 'kube_deployment_status_replicas_available{namespace="bookshop-staging"}'
1  deployment=bookshop

Argo CD: Synced Healthy.
Kubernetes: one pod available, as wanted.
The page: answering, 200, so bookshop-up passed.
The banner check: failed, with probe_failed_due_to_regex 1: it answered, and the words weren't there.

Every check from Argo CD Lesson 9 said fine. One check, written by someone who knew what the page should say, didn't. That's the only kind of check that catches "works, and is wrong". Then the fix, the GitOps way, as in Lesson 9:

OUTPUT

merged: 77bf3a8 Revert "Staging: a new banner (#46)" (#47)
deployment "bookshop" successfully rolled out

the banner check passed again after about 20s

$ q 'avg_over_time(probe_success{job="bookshop-banner"}[15m])'
0.8333333333333334  instance=http://bookshop.bookshop-staging.svc/
1  instance=http://bookshop.bookshop.svc/

Back in 20 seconds. And avg_over_time over 15 minutes: 0.83. probe_success is 1 or 0, so its average is the share of checks that passed: staging's page was right for 83% of the last 15 minutes, production's for all of it. That's how availability is measured, and how a promise like "99.9% of checks pass each month" is checked.

In real systems

Check from where customers are. Here, the blackbox exporter is in the same cluster as the shop, so it can't see a broken load balancer, an expired certificate or a DNS problem outside. Teams run checks from outside too: a blackbox exporter elsewhere, or a hosted service checking from several countries. Same idea, real network.
Check the certificate. For HTTPS addresses, probe_ssl_earliest_cert_expiry says when the certificate expires. "Expires in under 14 days" is one of the most useful alerts there is.
Content checks need care. Match something stable that proves the page works, such as a product list or a price, not text that changes every week. This lesson's banner check would fail the day marketing changed the banner, which is a test of the check as much as of the page.
Synthetic monitoring is the general name: scripted visits, from logging in to finishing a purchase, run on a timer. A blackbox HTTP check is its simplest form.

Spot the problem

Work out what's wrong before you read the explanation.

The alert that never fired. A team adds a blackbox check for their checkout page, and an alert: "fire if up is 0 for the checkout probe for 5 minutes". Weeks later, a release breaks checkout. Every request returns an error page, for two hours, until a customer emails. The alert never fired. Prometheus was running, the probe was configured, and the blackbox exporter was answering. What went wrong? They alerted on up, not probe_success. For a probe, up says whether Prometheus could reach the blackbox exporter, and it always could. The exporter's answer was "the check failed", every 30 seconds, in probe_success 0, for two hours, and nothing was watching that. The fix is one word: probe_success == 0 instead of up == 0. The habit to take from it: for every alert, make it fire once on purpose, as this lesson did with the missing page and the typo banner, before trusting it to fire on its own.

Keep this

To	Use
Install the blackbox exporter	An Application for `prometheus-blackbox-exporter`, pinned
Add a content check	A module with `fail_if_body_not_matches_regexp`
Ask by hand	`curl 'localhost:9115/probe?module=MODULE&target=URL'`
See why it failed	Add `&debug=true` to that address
Check a Service in the cluster	`http://SERVICE.NAMESPACE.svc/`
Which Probes Prometheus takes	`kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.probeSelector}'`
Check on a timer, from Git	A Probe: `prober`, `module`, `targets`, and the `release` label
Did the check pass	`probe_success`: 1 or 0. Not `up`
Why it failed	`probe_http_status_code`, `probe_failed_due_to_regex`
Where the time went	`probe_http_duration_seconds`, by `phase`
Share of checks that passed	`avg_over_time(probe_success[1h])`

Where you are now

The bookshop is checked from both sides now: nginx's own counts from the inside, and a visitor's view from the outside, including whether the page says what it should. Together they catch a pod that won't start, a page that won't load, and a release that loads and is wrong. Lesson 6 puts all of it on a Grafana dashboard, kept in Git like everything else.

Check yourself

What's the difference between white-box and black-box monitoring?
What is a module, in the blackbox exporter?
The missing page's probe failed. What was up for it?
With the typo banner on staging, which check failed?
What did probe_http_duration_seconds by phase show for the shop?
avg_over_time(probe_success[15m]) said 0.83 for staging. What does that mean?

Key terms

White-box monitoring: checking from the inside, with what a system reports about itself.
Black-box monitoring: checking from the outside, as a user would.
Blackbox exporter: checks an address on request and reports the result as metrics.
Module: a blackbox exporter setting saying what kind of check to do.
Content check: a check that the answer contains the right thing, not just that there was an answer.
Probe: a resource telling Prometheus to run blackbox checks on a timer.
Availability: the share of time, or of checks, that something worked.
Synthetic monitoring: scripted visits run on a timer, to check a service the way users would.

Go deeper

The blackbox exporter (opens in a new tab), especially its example modules (opens in a new tab), and the Prometheus Operator's Probe documentation.
Try: add a module that checks the shop's page is fast as well as right: http_2xx with a short timeout. What's the smallest timeout that still passes, and why is that a fragile thing to alert on?
Try: add http://guestbook-ui.guestbook.svc/ to bookshop-up's targets, through a pull request. Does it pass, and how big is its page?