Kubernetes basics · Lesson 3 of 12 · about 25 minutes

Pods: the smallest piece

What a pod really is, how to create one, how to look inside it, and what it looks like when one breaks.

By the end you'll know

  • What a pod is, and why Kubernetes doesn't just run containers directly
  • How to create a pod from a file, look inside it, and read its logs
  • What the two most common pod failures look like, and how to diagnose them
  • Why you almost never create pods by hand

What a pod is

In Lesson 2 you saw pods everywhere, even the control plane runs as pods. A pod is the smallest thing Kubernetes runs. It's a wrapper around one or more containers that:

  • share one network address. The pod gets a single IP address, and containers inside it reach each other on localhost. That also means they share ports: two containers in one pod can't both listen on port 80.
  • can share storage. A folder (called a volume) can be mounted into more than one container in the pod.
  • always run together, on the same node. The scheduler places the whole pod, never half of it.
Think of an apartment. The containers are roommates. They share one street address and the same kitchen, and they move in and out together. Most apartments have one tenant; that's the usual pod with one container.
What is inside a podOne node holds two pods. The first pod, called web, has one IP address and two containers: an nginx web server and a log shipper helper, called a sidecar. They talk to each other over localhost and both use a shared volume. The second pod, called api, has its own IP address and a single container, which is the most common case.Node: one machinePod: webone IP address: 10.244.1.5Container: webnginx, port 80Container: logsa "sidecar" helperlocalhostshared volume: /var/log/nginxPod: apiits own IP address: 10.244.1.6Container: apione container: the usual caseContainers in one pod share an address and storage, and always run on the same node.
Two pods on one node. The web pod has a main container plus a sidecar that ships its logs; they share an IP address and a volume. The api pod is the common case: one pod, one container.

When would you put two containers in one pod? Only when they truly can't live apart, like a helper that reads the main app's log files. If two things could run on different machines, they belong in different pods.

Try it: your first pod

You need a cluster again. Either one from Lesson 2 works; the single-node version is plenty:

BASH
kind create cluster --name learn

Save this as pod.yaml:

YAML
apiVersion: v1          # which version of the Kubernetes API this uses
kind: Pod               # what you're creating
metadata:
  name: hello           # the pod's name, used in kubectl commands
  labels:
    app: hello          # a tag; other objects use labels to find pods (Lesson 8)
spec:                   # what should be inside
  containers:
    - name: web                # the container's name inside the pod
      image: nginx:1.27        # the image to download and run
      ports:
        - containerPort: 80    # the port the app listens on

You'll read files like this line by line in Lesson 4.

Send it to the cluster and check on it:

BASH
kubectl apply -f pod.yaml
kubectl get pods
TEXT
NAME    READY   STATUS    RESTARTS   AGE
hello   1/1     Running   0          15s

READY 1/1 means one of one containers is ready. If you're quick you may catch it in ContainerCreating while the image downloads.

Look inside

Where is it, and what's its address?

BASH
kubectl get pod hello -o wide

The IP column is the pod's own address inside the cluster, and NODE is where the scheduler put it.

What happened, step by step?

BASH
kubectl describe pod hello

Scroll to the bottom, to Events:

TEXT
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  40s   default-scheduler  Successfully assigned default/hello to learn-control-plane
  Normal  Pulling    39s   kubelet            Pulling image "nginx:1.27"
  Normal  Pulled     33s   kubelet            Successfully pulled image "nginx:1.27"
  Normal  Created    33s   kubelet            Created container: web
  Normal  Started    33s   kubelet            Started container web

That's Lesson 2 happening in front of you: the scheduler picked a node, then that node's kubelet pulled the image and started the container. When a pod misbehaves, describe and its events are the first place to look.

Is it actually working? Forward a port from your laptop to the pod:

BASH
kubectl port-forward pod/hello 8080:80

Open http://localhost:8080 in your browser and you'll see "Welcome to nginx!". Press Ctrl+C to stop forwarding.

What has it been doing?

BASH
kubectl logs hello

You'll see a line for the page you just loaded. Logs are whatever the container prints, which is why apps in containers write their logs to the screen instead of to files.

Run a command inside it:

BASH
kubectl exec hello -- nginx -v

Break it on purpose

Pods fail in a handful of recognizable ways. Two of them cover most of what you'll see.

1. A bad image name. Start a pod whose image doesn't exist:

BASH
kubectl run typo --image=nginx:not-a-real-tag
kubectl get pods
TEXT
NAME    READY   STATUS             RESTARTS   AGE
typo    0/1     ImagePullBackOff   0          30s

ErrImagePull, then ImagePullBackOff: the kubelet can't download the image, so it keeps retrying, waiting longer each time. kubectl describe pod typo tells you exactly which image failed and why. The fix is almost always a typo in the name or tag, or a private image the cluster isn't allowed to pull.

2. An app that keeps crashing. Save this as crashy.yaml. The container prints two lines and exits with an error:

YAML
apiVersion: v1
kind: Pod
metadata:
  name: crashy
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "echo starting up; sleep 3; echo something went wrong; exit 1"]
BASH
kubectl apply -f crashy.yaml
kubectl get pods --watch

Watch for a minute (Ctrl+C to stop). The status flips between Error and CrashLoopBackOff, and RESTARTS keeps climbing. The kubelet restarts a crashed container automatically, waiting longer between tries, up to five minutes. To see why it died, read the logs from the attempt that crashed:

BASH
kubectl logs crashy --previous

CrashLoopBackOff is not the problem itself. It means "this keeps dying, and I'm pausing before the next try". The real reason is in the logs, or under Last State in kubectl describe.

A third failure is worth knowing by sight. When a container uses more memory than it's allowed, kubectl describe pod shows this under the container's Last State:

TEXT
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

OOMKilled means "out of memory, killed": the container went over its memory limit (Lesson 9) and was stopped. The kubelet restarts it, it runs out of memory again, and you're in CrashLoopBackOff.

In my real cluster

This is exactly what happened to my Prometheus monitoring in October. It kept running out of memory and sat in CrashLoopBackOff for 9 days, with 2,012 restarts. A restart count in the thousands is one of the loudest signals Kubernetes gives you, if something is watching for it.

Why you rarely create pods yourself

Delete the first pod:

BASH
kubectl delete pod hello
kubectl get pods

It's gone, and nothing brings it back. In Lesson 2, when you deleted a pod, a new one appeared. The difference: that pod belonged to a Deployment, whose controller kept "two copies" true. This pod was on its own.

A bare pod gets its crashed containers restarted by the kubelet, but if the pod itself is deleted, or its node dies, it's gone for good. That's why real apps are almost always run through a Deployment, which creates and replaces pods for you. That's Lesson 5, which also covers the other ways to run pods, such as one copy on every node, or a task on a schedule.

Clean up

BASH
kind delete cluster --name learn

Check yourself

  1. Two containers in the same pod both try to listen on port 8080. What happens?

  2. A new pod shows ImagePullBackOff. What should you do first?

  3. A pod is in CrashLoopBackOff. Which command shows why the last attempt died?

  4. A container shows Reason: OOMKilled and Exit Code: 137. What happened?

  5. You delete a pod you created from pod.yaml. Does it come back?

  6. When should two containers share a pod?

Key terms

  • Pod: one or more containers that share a network address and storage, and always run together on one node.
  • Sidecar: a helper container in the same pod as the main app, such as a log shipper.
  • Events: the step-by-step history at the bottom of kubectl describe. Your first stop when something's wrong.
  • ImagePullBackOff: the node can't download the container image and keeps retrying.
  • CrashLoopBackOff: the container keeps crashing, and Kubernetes is waiting before restarting it again.
  • OOMKilled: the container used more memory than its limit allows and was stopped.

Go deeper