Health checks: ready, alive and restarted

The problem: running isn't working

Back in Lesson 3, a pod's status told you whether its container was running. But Kubernetes only knows that the program hasn't exited. It doesn't know whether the app inside is actually doing its job. An app can be:

• Still starting up: the process is running, but it hasn't loaded its settings or connected to its database yet. Send it a request now and it fails.
• Stuck: the process is running, but it has frozen and stopped answering. It will never recover on its own.

From the outside, both look exactly like a healthy pod: Running. Health checks, which Kubernetes calls probes, fix that. You tell Kubernetes how to ask your app "are you OK?", and it asks on a schedule.

Three questions

There are three probes. Each asks a different question, and Kubernetes does something different when the answer is no:

• Readiness: "Can you take requests right now?" If not, Kubernetes stops sending the pod traffic, but leaves it running. When it passes again, traffic comes back. Nothing is restarted.
• Liveness: "Are you still working at all?" If not, Kubernetes restarts the container. This is for apps that are stuck and won't recover on their own.
• Startup: "Have you finished starting?" Only for apps that take a long time to start. Until it passes, the other two probes wait, so a slow start isn't mistaken for a stuck app.

How Kubernetes asks

A probe is a few lines inside a container's spec. You choose how Kubernetes asks, and how often:

readinessProbe:
  httpGet:              # send an HTTP request...
    path: /             # ...to this path...
    port: 80            # ...on this port. Any 200 to 399 reply counts as a pass.
  periodSeconds: 5      # ask every 5 seconds
  failureThreshold: 3   # 3 failures in a row before acting

httpGet is the most common way to ask, and the one this lesson uses. There are two others: tcpSocket only checks that a port accepts connections, and exec runs a command inside the container, where exit code 0 means healthy.

failureThreshold matters more than it looks. One slow reply shouldn't restart your app, so Kubernetes waits for several failures in a row before it acts.

Try it: an app with health checks

Start a cluster:

kind create cluster --name learn

Save this as web.yaml. It's the same kind of Deployment as Lesson 5, now with two probes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.28
          ports:
            - containerPort: 80
          readinessProbe:          # can it take requests?
            httpGet:
              path: /
              port: 80
            periodSeconds: 5
          livenessProbe:           # is it still working?
            httpGet:
              path: /
              port: 80
            periodSeconds: 10
            failureThreshold: 3

kubectl apply -f web.yaml
kubectl get pods

Look at the READY column. Each pod shows 0/1 for a few seconds, then 1/1. That's the readiness probe: the container started almost immediately, but the pod only counts as ready once nginx actually answered on /.

Watch readiness save a bad release

Readiness matters most during an update. In Lesson 5, a rolling update started a new pod, waited for it to be ready, and only then stopped an old one. With a readiness probe, "ready" means "the app really answers", not just "the container started".

Ship a release that doesn't work. In web.yaml, change the readiness probe's path from / to /ready, a page nginx doesn't have, and apply:

kubectl apply -f web.yaml
kubectl get pods

You'll see the three old pods still 1/1, plus one new pod that's Running but stuck at 0/1. Every request it gets for /ready returns 404 Not Found, so it never becomes ready. And because it never becomes ready, the rollout never moves on: no old pod is stopped. Ask why:

kubectl get events --field-selector reason=Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 404. The bad version never took a single request, and the old version kept running the whole time. Roll it back, as in Lesson 5:

kubectl rollout undo deployment/web

Then change the path in web.yaml back to /, so the next apply doesn't bring the broken version back.

In Lesson 8 you'll meet Services, which send requests to pods. They only send requests to pods that are ready, which is how a readiness probe keeps a starting or struggling pod out of the way.

Break it on purpose

Now break the liveness probe instead. In web.yaml, keep readiness on /, and change the liveness probe's path to /alive. Apply, and watch:

kubectl apply -f web.yaml
kubectl get pods -w

The new pods become ready, because readiness still passes, and the rollout finishes. Then, about 30 seconds later (3 failures, 10 seconds apart), the RESTARTS count starts climbing. nginx is fine and serving pages. Kubernetes is restarting it because you told it to check a page that doesn't exist. Keep watching, and the pods eventually go into CrashLoopBackOff (Lesson 3), with longer and longer waits between restarts. Press Ctrl+C, then:

kubectl get events --field-selector reason=Unhealthy

Liveness probe failed: HTTP probe failed with statuscode: 404. Run kubectl get events without the filter and you'll also see the kubelet restarting the container because of it. That's the danger of liveness probes: a wrong one turns a healthy app into a crashing one. Two rules of thumb:

• Liveness should only check the app itself. Never make it depend on a database or another service, or one outage restarts everything.
• If you're not sure an app needs a liveness probe, leave it out. Kubernetes already restarts containers that exit. A liveness probe is for apps that can freeze without exiting.

Change the path back to / and apply again to fix it.

Clean up

kind delete cluster --name learn

Check yourself

Key terms

• Probe: a health check Kubernetes runs on a container, on a schedule.
• Readiness probe: asks whether the container can take requests right now. Failing it removes the pod from traffic, without a restart.
• Liveness probe: asks whether the container is still working. Failing it restarts the container.
• Startup probe: asks whether a slow app has finished starting. The other probes wait until it passes.
• failureThreshold: how many failures in a row before Kubernetes acts.
• httpGet, tcpSocket, exec: the ways a probe can ask: an HTTP request, a connection to a port, or a command inside the container.

Go deeper

• Liveness, readiness and startup probes (opens in a new tab) in the official docs.
• Configure liveness, readiness and startup probes (opens in a new tab) walks through every setting, with examples.