Storage: keeping data when pods restart

The problem: containers forget

When a container restarts, it starts fresh from its image. Anything it wrote to its own files since it started is gone. For most apps that's fine, even good: a web server or an API keeps nothing it can't rebuild. But some apps exist to keep data: a database, a file upload service, a metrics store. For those, Kubernetes needs a way to keep files around longer than one container.

That's what volumes are: storage that's attached to a pod from outside the container. The question is always how long the data needs to last.

Three lifetimes

An emptyDir is an empty folder created when the pod starts, and deleted when the pod goes away. It belongs to the pod, not the container, so it survives a container restart. Remember that Deployments replace pods all the time: during updates, when a node is drained, when you delete one. Every replacement starts with a new, empty emptyDir.

A PersistentVolume lives separately from any pod. Pods come and go; the volume stays, and the next pod picks up where the last one left off.

How a pod asks for storage

Storage on a cloud is a real disk somewhere, and a pod shouldn't need to know which one. So Kubernetes splits it into three pieces:

• PersistentVolumeClaim (PVC): the pod's request. "I need 1 gigabyte, that one node can read and write." You write this.
• StorageClass: how this cluster creates storage. On DigitalOcean it creates a block storage disk; on AWS, an EBS disk; on kind, a folder on your computer. Someone sets this up once per cluster.
• PersistentVolume (PV): the actual storage. The StorageClass creates one to match each claim, and binds the two together.

Two more words you'll see in a claim:

• Access mode: who can use the volume at once. The common one is ReadWriteOnce: read and write, from one node at a time. ReadWriteMany lets pods on many nodes share it, but only some kinds of storage support that.
• Reclaim policy: what happens to the storage when you delete the claim. Storage created by a StorageClass usually has Delete: the claim goes, and the data goes with it. Retain keeps the data until someone removes it by hand.

Try it: data that outlives its pod

Start a cluster:

kind create cluster --name learn
kubectl get storageclass

kind comes with one StorageClass, standard, marked (default). Its PROVISIONER, rancher.io/local-path, creates each volume as a folder on the node. That's the local-path-storage namespace you saw in Lesson 10. A claim that doesn't name a StorageClass gets the default one.

Save this as claim.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: notes-data
spec:
  accessModes:
    - ReadWriteOnce        # read and write, from one node at a time
  resources:
    requests:
      storage: 1Gi         # how much space to ask for

kubectl apply -f claim.yaml
kubectl get pvc

STATUS: Pending. That's normal here, not an error. kind's StorageClass waits until a pod actually uses the claim before it creates the storage, so the storage ends up on the same node as the pod. Now give it a pod. Save this as notes.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["sleep", "3600"]
          volumeMounts:
            - name: data
              mountPath: /data          # the volume appears here inside the container
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: notes-data       # use the claim, not a specific disk

kubectl apply -f notes.yaml
kubectl get pvc
kubectl get pv

Once the pod is running, the claim is Bound, and there's a PersistentVolume with a generated name like pvc-3f2a... that nobody had to write. That's the StorageClass at work. Now write something to the volume:

kubectl exec deploy/notes -- sh -c 'echo "written before the pod was deleted" > /data/note.txt'

Delete the pod. The Deployment replaces it with a brand-new one:

kubectl delete pod -l app=notes
kubectl get pods

When the new pod is Running, read the file:

kubectl exec deploy/notes -- cat /data/note.txt

Still there. Go further, and delete the whole Deployment, then create it again:

kubectl delete deployment notes
kubectl apply -f notes.yaml

Give the new pod a few seconds to start, then:

kubectl exec deploy/notes -- cat /data/note.txt

Still there. The claim and its volume don't belong to any pod or Deployment, so they outlive both.

Break it on purpose

Ask for a StorageClass that doesn't exist. Save this as bad-claim.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fast-data
spec:
  storageClassName: fast-ssd   # no StorageClass by this name
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

kubectl apply -f bad-claim.yaml
kubectl get pvc
kubectl describe pvc fast-data

fast-data stays Pending, and this time it will stay that way forever. The Events at the bottom of describe say why: the StorageClass fast-ssd wasn't found. Any pod that uses this claim would also wait in Pending, because Kubernetes won't start a pod whose storage doesn't exist.

When a claim is stuck in Pending, check the StorageClass name against kubectl get storageclass. And remember the first Pending you saw: on kind, a claim waits for its first pod, so a claim with no pod yet isn't a problem. describe tells you which case you're in.

Clean up

kind delete cluster --name learn

Check yourself

Key terms

• Volume: storage attached to a pod from outside the container.
• emptyDir: a volume that starts empty and lasts as long as its pod. Survives container restarts, not pod replacement.
• PersistentVolume (PV): storage that lives separately from any pod.
• PersistentVolumeClaim (PVC): a request for storage: how much, and how it may be used. Pods name claims, not volumes.
• StorageClass: how a cluster creates storage on request. Most clusters have a default one.
• Access mode: who can use a volume at once. ReadWriteOnce (one node) is the most common.
• Reclaim policy: what happens to the storage when its claim is deleted: Delete or Retain.

Go deeper

• Volumes (opens in a new tab) in the official docs, including emptyDir.
• Persistent Volumes (opens in a new tab) covers claims, access modes and reclaim policies in depth.
• Storage Classes (opens in a new tab) explains how clusters create storage on request.