What monitoring solves, and a first look

By the end you'll know

What monitoring answers, and the two things it can't see
The pieces of a Prometheus stack, and what each one does
How to install it with Argo CD, from the kube-prometheus-stack chart
How to read Prometheus's targets and alerts, through its API
What Kubernetes already tells Prometheus about an app, and what it doesn't
Why a chart's generated password changes under Argo CD, and how to fix it

Before you start

You need: the Argo CD and Helm track done, with the gitops cluster running, all Applications Synced, and Sealed Secrets installed (Lesson 10).
Room for it: this stack is the biggest thing the cluster has run. Docker had 16 CPUs and about 8 GB of memory here, which was enough. Check yours in Docker Desktop's settings.
Install jq if you don't have it: brew install jq. Prometheus answers in JSON, and jq picks out the parts you want.
Where commands run: your own computer's terminal, in gitops-practice, with the Argo CD port-forward running and argocd login done.

The short version

The install. One Application, the kube-prometheus-stack chart at version 92.18.0. Synced and Healthy in 182 seconds: 9 pods and 132 resources.
What it watches. 24 targets, 19 up. The 5 down are parts of Kubernetes that kind doesn't let Prometheus reach.
Alerts on a healthy cluster. 155 alert rules. Within 10 minutes, 4 alerts were firing on a cluster where nothing was wrong, plus one that always fires, on purpose.
The bookshop. Prometheus already knows its pods, restarts, memory and CPU. It knows nothing about the requests it serves.
A surprise. Grafana's password changed every time Argo CD rendered the chart afresh, overnight too, with nobody touching anything. A sealed password of your own fixed it.

What monitoring answers

The Argo CD track kept asking questions it could only answer by looking: is the shop up? Did that rollout drop any requests? Is the new release working? Each time, the answer came from a command run at that moment, like a probe pod counting replies. Monitoring asks those questions all the time, keeps the answers, and tells someone when an answer goes wrong.

It answers three kinds of question:

Is it up? Is each part running and answering?
Is it healthy? How busy is it, how much memory, how many errors, how slow?
Is it about to break? Is a disk filling, memory creeping, are errors rising?

And it has two blind spots, which this track comes back to:

Whether it's right. Lesson 9 of the Argo CD track deployed a banner full of typos, and everything was Healthy. Numbers can't tell you a page is wrong, unless something checks for what right looks like.
Itself. If the monitoring stops, it can't tell you it stopped. Silence looks exactly like everything being fine. Lesson 10 deals with that.

The pieces

The usual open-source answer is Prometheus and the tools around it:

Prometheus collects metrics: numbers with names, such as how many requests an app has served, recorded every few seconds. It pulls them: every 30 seconds, by default here, it asks each target for its current numbers. Asking is called scraping.
Exporters turn things that don't speak Prometheus into targets. node-exporter reports on each machine: CPU, memory, disk. kube-state-metrics reports on Kubernetes objects: how many pods a Deployment wants and has.
Alertmanager receives alerts from Prometheus and decides who hears about them, and when.
Grafana draws dashboards from Prometheus's numbers.
The Prometheus Operator runs all of it inside Kubernetes. It adds new kinds of resources, such as a ServiceMonitor, which tells Prometheus "scrape the pods behind this Service", so what gets monitored is described in YAML, like everything else.

kube-prometheus-stack is a Helm chart that installs the lot, already wired together, with a few hundred alert rules and dashboards written by people who run Kubernetes for a living. That's a lot to take on trust, which is why this lesson looks at what it brought.

Installing it with Argo CD

The chart lives in the prometheus-community chart repository:

BASH

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm search repo prometheus-community/kube-prometheus-stack

OUTPUT

NAME                                      	CHART VERSION	APP VERSION	DESCRIPTION
prometheus-community/kube-prometheus-stack	92.18.0      	v0.94.2    	kube-prometheus-stack collects Kubernetes manif...

The same pattern as Sealed Secrets in Lesson 10: one Application in argocd/, naming the chart and pinning its version, for the root to pick up.

YAML

# Monitoring (Monitoring track, Lesson 1): Prometheus, Alertmanager, Grafana and the
# exporters that feed them, from the kube-prometheus-stack Helm chart.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: monitoring
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    chart: kube-prometheus-stack
    targetRevision: 92.18.0
    helm:
      releaseName: monitoring
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      # Some of the chart's definitions are too large for the older way of applying,
      # as with Argo CD's own install in the Argo CD track's Lesson 1.
      - ServerSideApply=true

Save it as argocd/monitoring.yaml. One new line: ServerSideApply=true. The chart's resource definitions are huge, too big for the way Argo CD applies things by default, so it applies them the way kubectl apply --server-side did when Argo CD itself was installed. Then, as for every app since Lesson 8, a pull request, and the root does the rest:

BASH

git switch -c monitoring
git add argocd/monitoring.yaml
git commit -m "Monitoring: kube-prometheus-stack, installed by Argo CD"
git push -u origin monitoring
gh pr create --fill --base main
gh pr merge --squash --delete-branch
git switch main && git pull

Merged, and checked every 45 seconds or so:

OUTPUT

merged: f2ccc82 Monitoring: kube-prometheus-stack, installed by Argo CD (#39)
after 42s: OutOfSync Progressing
after 89s: OutOfSync Progressing
after 136s: OutOfSync Degraded
after 182s: Synced Healthy

Degraded at 136 seconds, then Healthy. A big chart arrives in pieces, with images downloading and parts waiting on each other. Argo CD reports each moment honestly, and the stack settled on its own. Remember that the next time something is briefly Degraded during an install: check again before acting.

What one Application installed

BASH

kubectl get pods -n monitoring

OUTPUT

NAME                                                     READY   STATUS      RESTARTS   AGE
alertmanager-monitoring-kube-prometheus-alertmanager-0   2/2     Running     0          62s
monitoring-grafana-5c589db88d-q55p6                      3/3     Running     0          15s
monitoring-kube-prometheus-admission-create-4qc2h        0/1     Completed   0          19s
monitoring-kube-prometheus-admission-patch-cxwln         0/1     Completed   0          2s
monitoring-kube-prometheus-operator-fbfdc479c-5vnhn      1/1     Running     0          2m43s
monitoring-kube-state-metrics-7994f48c57-fw69v           1/1     Running     0          2m43s
monitoring-prometheus-node-exporter-sdtsm                1/1     Running     0          2m43s
monitoring-prometheus-node-exporter-smhnp                1/1     Running     0          2m43s
prometheus-monitoring-kube-prometheus-prometheus-0       2/2     Running     0          62s

Every piece from above: the operator, Prometheus, Alertmanager, Grafana, kube-state-metrics, and one node-exporter on each of the two nodes. The two admission pods ran once, set something up, and finished. Note the Grafana pod: 15 seconds old, when everything else is a minute or more. That comes back at the end of the lesson. And everything else the chart made, counted by kind:

BASH

argocd app resources monitoring | awk 'NR>1{print $2}' | sort | uniq -c | sort -rn

OUTPUT

  47 monitoring
  35 PrometheusRule
  13 ServiceMonitor
  10 CustomResourceDefinition
   5 kube-system
   5 ClusterRoleBinding
   5 ClusterRole
   3 Deployment
   2 Job
   1 ValidatingWebhookConfiguration
   1 RoleBinding
   1 Role
   1 Prometheus
   1 MutatingWebhookConfiguration
   1 DaemonSet
   1 Alertmanager

132 resources, from one file in Git. (Run it again a day later and it's 124: the chart's one-off setup jobs, and the permissions they used, clean themselves up once they've succeeded, and the fix at the end of this lesson removes one more.) (The monitoring and kube-system lines are the column for each resource's namespace, which awk picked up for the ones without a group: Services, ConfigMaps, Secrets and the like.) 35 PrometheusRules hold the alert rules; 13 ServiceMonitors say what to scrape. And 10 CustomResourceDefinitions: new kinds of resource, the operator's vocabulary:

OUTPUT

alertmanagerconfigs.monitoring.coreos.com
alertmanagers.monitoring.coreos.com
podmonitors.monitoring.coreos.com
probes.monitoring.coreos.com
prometheusagents.monitoring.coreos.com
prometheuses.monitoring.coreos.com
prometheusrules.monitoring.coreos.com
scrapeconfigs.monitoring.coreos.com
servicemonitors.monitoring.coreos.com
thanosrulers.monitoring.coreos.com

There's even a resource called Prometheus: one of them, describing the Prometheus server itself. The operator reads it and runs the pod. You won't write one; you'll write ServiceMonitors and PrometheusRules, starting in Lesson 4.

What Prometheus is watching

Prometheus has a web interface and an API, both on port 9090. A port-forward, as with Argo CD, in its own terminal:

BASH

kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090

http://localhost:9090 in a browser shows it, and Status, Targets lists everything it scrapes. The same list from the API, counted by health and job:

BASH

curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | "\(.health) \(.labels.job)"' | sort | uniq -c

OUTPUT

   1 down kube-controller-manager
   1 down kube-etcd
   2 down kube-proxy
   1 down kube-scheduler
   1 up apiserver
   2 up coredns
   1 up kube-state-metrics
   6 up kubelet
   1 up monitoring-grafana
   2 up monitoring-kube-prometheus-alertmanager
   1 up monitoring-kube-prometheus-operator
   2 up monitoring-kube-prometheus-prometheus
   2 up node-exporter

24 targets, 19 up. The kubelet on each node (three targets each: the kubelet, its container stats, and its health checks), the API server, DNS, the exporters, and the monitoring pieces watching each other. And why the five are down:

OUTPUT

kube-controller-manager  https://172.18.0.2:10257/metrics  ... connect: connection refused
kube-etcd  http://172.18.0.2:2381/metrics  ... connect: connection refused
kube-proxy  http://172.18.0.2:10249/metrics  ... connect: connection refused
kube-proxy  http://172.18.0.3:10249/metrics  ... connect: connection refused
kube-scheduler  https://172.18.0.2:10259/metrics  ... connect: connection refused

Connection refused, from Kubernetes' own control-plane parts. On kind, those listen only on their own node's localhost, so nothing else can reach them, including Prometheus. Nothing is broken; the chart expects a cluster set up to let them be scraped, and this one isn't. A managed cluster, like EKS, usually hides them altogether. Either way, the chart's defaults assume a cluster that isn't yours.

Alerts, on a cluster where nothing is wrong

BASH

curl -s http://localhost:9090/api/v1/rules | jq '[.data.groups[].rules[] | select(.type=="alerting")] | length'

OUTPUT

155 alert rules came with the chart. Minutes after the install, which were going off:

OUTPUT

   1 firing none Watchdog
   1 pending critical etcdInsufficientMembers
   1 pending warning etcdMembersDown
   1 pending warning KubeControllerManagerInstanceUnreachable
   2 pending warning KubeProxyInstanceUnreachable
   1 pending warning KubeSchedulerInstanceUnreachable
   2 pending warning NodeClockNotSynchronising
   4 pending warning TargetDown

And 13 minutes later:

OUTPUT

   1 firing critical etcdInsufficientMembers  since 02:33:18
   1 firing none Watchdog  since 02:32:45
   2 firing warning NodeClockNotSynchronising  since 02:33:10
   4 firing warning TargetDown  since 02:33:15
   1 pending warning etcdMembersDown  since 02:33:18
   1 pending warning KubeControllerManagerInstanceUnreachable  since 02:33:26
   2 pending warning KubeProxyInstanceUnreachable  since 02:33:13
   1 pending warning KubeSchedulerInstanceUnreachable  since 02:33:18

(The times are UTC, Prometheus's clock.) An alert is pending when its condition is true, and firing once it has stayed true for long enough. Two of the rules:

OUTPUT

TargetDown: for 600s
  100 * (count by (cluster, job, namespace, service) (up == 0) / count by (cluster, job, namespace, service) (up)) > 10
NodeClockNotSynchronising: for 600s
  min_over_time(node_timex_sync_status{job="node-exporter"}[5m]) == 0 and node_timex_maxerror_seconds{job="node-exporter"} >= 16

for: 600s: true for 10 minutes before it fires, so a blip doesn't wake anyone. Those lines underneath are PromQL, Prometheus's query language, Lesson 3's subject. TargetDown says "more than 10% of a job's targets are down", and fired because of the five unreachable targets. etcdInsufficientMembers is critical, the most urgent level, and is the same five targets seen another way. NodeClockNotSynchronising is real: kind's nodes don't keep their clocks in step with a time server. On a real machine it would matter; on a laptop cluster it doesn't.

Watchdog is different. It fires always, on purpose: an alert whose only job is to prove the alerting works. If it ever stops arriving, the monitoring has stopped. That's Lesson 10's dead-man's switch.

So, five firing alerts, and nothing wrong. This is what a fresh install looks like, and the real lesson of it: an alert that fires for something nobody needs to act on teaches people to ignore alerts. Lesson 7 turns off the rules that don't fit this cluster, in Git.

Grafana

Grafana is on port 80 of its Service, so another port-forward, then http://localhost:3000:

BASH

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80
curl -s http://localhost:3000/api/health | jq .

OUTPUT

{
  "database": "ok",
  "version": "13.3.0",
  "commit": "6f8ce3c693b493043ebcd4787566d2b6b118b0f0"
}

Grafana 13.3.0. Its API needs the admin login. At this point the chart has made up a password and kept it in a Secret, so read it from there:

BASH

GPW=$(kubectl get secret monitoring-grafana -n monitoring -o jsonpath='{.data.admin-password}' | base64 -d)
echo "${#GPW} characters"

OUTPUT

40 characters

(If you've already done the fix at the end of this lesson, that Secret is gone: use GPW=$(cat .grafana-admin) instead.) Then its dashboards and data sources:

OUTPUT

$ curl -s -u admin:"$GPW" 'http://localhost:3000/api/search?type=dash-db' | jq length
29

$ curl -s -u admin:"$GPW" http://localhost:3000/api/datasources | jq -r '.[] | "\(.name) \(.type) \(.url)"'
Alertmanager alertmanager http://monitoring-kube-prometheus-alertmanager.monitoring:9093/
Prometheus prometheus http://monitoring-kube-prometheus-prometheus.monitoring:9090/

29 dashboards, ready-made: Kubernetes / Compute Resources / Pod, Node Exporter / Nodes, Prometheus / Overview and more. Two data sources, Prometheus and Alertmanager, found by their Services' names inside the cluster. That made-up password turned out to be a problem, below.

What it can see about the bookshop

Nothing was added to the bookshop. What does Prometheus know about it anyway?

BASH

q(){ curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=$1"; }
q 'kube_deployment_status_replicas_available{namespace="bookshop"}' | jq -r '.data.result[] | "\(.metric.deployment) \(.value[1])"'
q 'kube_pod_container_status_restarts_total{namespace="bookshop"}' | jq -r '.data.result[] | "\(.metric.pod) \(.value[1])"'

OUTPUT

bookshop 2

bookshop-74d9f67bc8-4kf4c 0
bookshop-74d9f67bc8-55btt 0

q is a short function that sends a PromQL query to the API. Two pods available, no restarts: kube-state-metrics, reading the Deployment and pods. And from the kubelet, what each pod uses:

OUTPUT

$ q 'sum by (pod) (container_memory_working_set_bytes{namespace="bookshop",container="nginx"})' ...
bookshop-74d9f67bc8-55btt 15355904
bookshop-74d9f67bc8-4kf4c 13750272

$ q 'sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="bookshop",container="nginx"}[5m]))' ...
bookshop-74d9f67bc8-55btt 0.000058515219370555356
bookshop-74d9f67bc8-4kf4c 0.0000576223686462433

About 14 MB of memory each, against the 64 MB limit from the chart, and almost no CPU: six hundred-thousandths of a core. How much, altogether?

OUTPUT

$ q 'count by (__name__) ({namespace="bookshop"})' | jq -r '.data.result | length'
125

$ q 'count by (job) ({namespace="bookshop"})' | jq -r '.data.result[] | "\(.metric.job) \(.value[1])"'
kubelet 376
kube-state-metrics 133
null 15

$ q '{namespace="bookshop",__name__=~"nginx.*|http.*"}' | jq '.data.result | length'
0

125 different metrics about the bookshop, and not one from the bookshop. All of them come from Kubernetes describing its pods from the outside: the kubelet and kube-state-metrics, plus a few that Prometheus works out for itself (the null job). Not one counts a request, an error or a slow page. Prometheus can see the bookshop is running. It can't see whether customers are getting pages. That's Lesson 4: the bookshop's own metrics.

The password that kept changing

That Grafana pod, younger than everything else, turned out to have company. The next morning:

OUTPUT

$ kubectl get rs -n monitoring -l app.kubernetes.io/name=grafana --sort-by=.metadata.creationTimestamp
NAME                            DESIRED   CURRENT   READY   AGE
monitoring-grafana-7d8778446    0         0         0       9h
monitoring-grafana-5c589db88d   0         0         0       9h
monitoring-grafana-769db985b9   0         0         0       9h
monitoring-grafana-59b4d85c5b   1         1         1       12m

Four ReplicaSets: Grafana's pod had been replaced three times, the last 12 minutes before this, overnight, with nobody touching anything. Comparing their pod templates, one annotation differed each time:

OUTPUT

--- replicaset.apps/monitoring-grafana-7d8778446
checksum/config=457c16ae5fdf9a6ab84c
checksum/secret=247d7f1f4cd9df2ab3e0

--- replicaset.apps/monitoring-grafana-5c589db88d
checksum/config=457c16ae5fdf9a6ab84c
checksum/secret=7558bfa4532c1aa78bb9

--- replicaset.apps/monitoring-grafana-769db985b9
checksum/config=457c16ae5fdf9a6ab84c
checksum/secret=f901ce32dca2f46086aa

checksum/secret, the same trick as the bookshop's checksum/page in the Argo CD track: a fingerprint of a file, so the pods roll when it changes. Here the file is Grafana's Secret, holding its admin password. The chart makes the password up, at random, every time it's rendered. Plain Helm avoids changing it by looking up the old one in the cluster first. Argo CD renders charts without asking the cluster anything, so every fresh render is a new password, a new Secret, a new checksum, and a new pod. Argo CD keeps rendered charts for a while, then renders again, which is why it happened overnight. A hard refresh, which throws the saved render away, does it on demand:

OUTPUT

$ argocd app get monitoring --hard-refresh -o json | jq -r '"\(.status.sync.status) \(.status.health.status)"'
OutOfSync Healthy

monitoring-grafana-59b4d85c5b   0         0         0       13m
monitoring-grafana-5858d6b659   1         1         1       20s

OutOfSync, synced by self-heal, and a fifth ReplicaSet 20 seconds later. Whoever logged in to Grafana would find their password no longer works.

The fix is Lesson 10's: a password of your own, sealed in Git. A random one, in a file Git ignores:

BASH

echo '.grafana-admin' >> .gitignore
openssl rand -base64 24 | tr -d '\n' > .grafana-admin && chmod 600 .grafana-admin

Sealed for the monitoring namespace, as a whole Secret this time, with two keys:

BASH

kubectl create secret generic grafana-admin -n monitoring \
  --from-literal=admin-user=admin --from-file=admin-password=.grafana-admin \
  --dry-run=client -o yaml | kubeseal -o yaml > apps/monitoring/grafana-admin.yaml

OUTPUT

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: grafana-admin
  namespace: monitoring
spec:
  encryptedData:
    admin-password: AgByren58Zgru6fivislyAkQmCLxGie1BwpwoBga...
    admin-user: AgAPBzC1eL52NEG6PrIVhB2zhK5ftP8yuvhrPyvU...

kubeseal -o yaml without --raw seals a whole Secret file into a SealedSecret file, ready to commit. A new folder, apps/monitoring/, holds it, with a small Application, monitoring-config, to apply that folder to the monitoring namespace. Dashboards and alert rules will go there too, in later lessons. Then the chart is told to use it, in the Application:

YAML

    helm:
      releaseName: monitoring
      valuesObject:
        grafana:
          admin:
            # Use the sealed login from apps/monitoring, rather than a password the
            # chart makes up again every time Argo CD renders it.
            existingSecret: grafana-admin
            userKey: admin-user
            passwordKey: admin-password

Two pull requests, #40 and #41: the sealed login first, so it exists before Grafana needs it. Then the test, two hard refreshes in a row:

OUTPUT

$ argocd app get monitoring --hard-refresh -o json | jq -r '"\(.status.sync.status) \(.status.health.status)"'
Synced Healthy

$ argocd app get monitoring --hard-refresh -o json | jq -r '"\(.status.sync.status) \(.status.health.status)"'
Synced Healthy

monitoring-grafana-5858d6b659   0         0         0       111s
monitoring-grafana-64595f4b85   1         1         1       58s

$ argocd app diff monitoring >/dev/null; echo "diff exit code: $?"
diff exit code: 0

Synced both times, and no new ReplicaSet. The one at 58 seconds is the switch to the new password itself. The chart's own Secret is gone, because it no longer has a password to keep, and the login works:

OUTPUT

$ curl -s -o /dev/null -w '%{http_code}\n' -u admin:"$(cat .grafana-admin)" http://localhost:3000/api/user
200

$ curl -s -o /dev/null -w '%{http_code}\n' -u admin:wrong-password http://localhost:3000/api/user
401

200 with the password, 401 without. Three files now stay out of Git: .argocd-admin, .sealed-secrets-key.yaml and .grafana-admin.

In real systems

Tune the defaults, in Git. Teams start from a chart like this and switch off the rules and targets that don't fit their cluster, in the chart's values. An alert nobody acts on is worse than no alert.
Look for generated values. Any chart that makes up a password or a key at render time behaves this way under Argo CD. The fix is always the same: supply the value yourself, from a Secret.
Prometheus keeps data on a disk. Here it's inside the pod and gone when the pod is replaced. Real setups give it a PersistentVolume, from the Kubernetes track's storage lesson, and decide how long to keep data.
One Prometheus doesn't scale forever. Large setups keep long-term data elsewhere, with tools such as Thanos or a hosted service, and use Prometheus for the last few hours or days.

Spot the problem

Work out what's wrong before you read the explanation.

The empty answer. The first time this lesson's script asked about the bookshop, it asked like this:

BASH

curl -s "http://localhost:9090/api/v1/query?query=up{namespace=\"bookshop\"}" | jq '.data.result | length'

OUTPUT

Zero: so no bookshop targets, which is true. The next queries, for the bookshop's pods, printed jq: error: Cannot iterate over null. What went wrong? The query never worked, not even the first one. Asking without jq shows the actual answer:

OUTPUT

{"status":"error","errorType":"bad_data","error":"invalid parameter \"query\": 1:12: parse error: unexpected \"=\""}

The braces, quotes and spaces in PromQL have to be encoded to travel in a web address, and pasted in raw, they arrived garbled. Prometheus refused with an error. curl doesn't treat an error from the server as a failure unless it's told to, with --fail, so nothing stopped. And jq turned "no result at all" into a 0 that looked exactly like "no bookshop targets". The fix is to let curl encode the query: curl -G --data-urlencode "query=...", as the q function does. The bigger lesson: check status before trusting an empty result. An error that looks like "nothing found" is worse than a loud one.

Keep this

To	Run
Install the stack with Argo CD	An Application for `kube-prometheus-stack`, pinned, with `ServerSideApply=true`
Reach Prometheus	`kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090`
Reach Grafana	`kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80`
Targets, by health	`curl -s localhost:9090/api/v1/targets \| jq -r '.data.activeTargets[] \| "(.health) (.labels.job)"' \| sort \| uniq -c`
Alerts, by state	`curl -s localhost:9090/api/v1/alerts \| jq -r '.data.alerts[] \| "(.state) (.labels.alertname)"' \| sort \| uniq -c`
Run a query	`curl -sG localhost:9090/api/v1/query --data-urlencode "query=QUERY"`
Check it worked	`jq .status`: `success`, or `error` with a reason
Grafana's health	`curl -s localhost:3000/api/health`
Throw away Argo CD's saved render	`argocd app get APP --hard-refresh`
Seal a whole Secret file	`kubectl create secret ... --dry-run=client -o yaml \| kubeseal -o yaml > FILE`
Give Grafana your own login	`grafana.admin.existingSecret`, `userKey`, `passwordKey` in the chart's values

Where you are now

The cluster watches itself: Prometheus scraping 24 targets, 155 alert rules, Grafana with 29 dashboards, all from Git, and a Grafana password that stays put. You've also seen its limits on day one: alerts for things nobody needs to fix, and nothing at all about what the bookshop's customers experience. Lesson 2 opens up a metric and looks at what it's made of: names, labels and the numbers underneath.

Check yourself

How does Prometheus get its numbers?
Five targets were down, with connection refused. What did that mean?
What's the difference between a pending and a firing alert?
Why does the Watchdog alert always fire?
Prometheus had 125 metrics about the bookshop. What was missing?
Why did Grafana's pod keep being replaced?

Key terms

Monitoring: collecting numbers about systems all the time, keeping them, and alerting when they go wrong.
Metric: a named number, recorded over time, such as memory used.
Target: something Prometheus scrapes.
Scrape: Prometheus asking a target for its current metrics.
Exporter: a program that turns something else's state into metrics, such as node-exporter for machines.
ServiceMonitor: a resource telling Prometheus which Service's pods to scrape.
PrometheusRule: a resource holding alert rules.
Pending, firing: an alert whose condition is true, and one that has stayed true for its for: time.
Hard refresh: Argo CD rendering an app again from scratch, instead of using its saved render.

Go deeper

The kube-prometheus-stack chart (opens in a new tab), especially its values.yaml, and the Prometheus documentation's overview (opens in a new tab).
Try: open http://localhost:9090 and look at Status, Targets and Alerts. Then in Grafana, open Kubernetes / Compute Resources / Namespace (Pods) and choose bookshop. What does it show, and what doesn't it?
Try: find TargetDown in the chart's rules with kubectl get prometheusrules -n monitoring -o yaml | grep -A8 'alert: TargetDown'. Which severity does it have, and what does its description say?