Requests, limits and choosing nodes

The problem: sharing machines

A node is a machine with a fixed amount of CPU and memory, shared by every pod on it. Two things can go wrong:

• The scheduler can't plan. In Lesson 2 you saw that it picks a node "with enough room". But room for what? If pods don't say how much they need, it's guessing.
• One greedy pod can starve the rest. A memory leak in one app can eat the whole node, and then everything on it suffers.

Kubernetes solves both with two numbers per container: a request and a limit.

Requests and limits

• Request: what the container is guaranteed. The scheduler adds up requests to decide where a pod fits, and only places it on a node with that much left unreserved.
• Limit: the most the container may ever use. It's a hard ceiling.

In YAML, they go on each container:

    spec:
      containers:
        - name: hello
          image: nginx:1.28
          resources:
            requests:
              cpu: 100m        # a tenth of one CPU core, reserved
              memory: 64Mi     # 64 mebibytes, reserved
            limits:
              memory: 128Mi    # never more than this

The units:

• CPU is measured in cores. 1 is one core, and 100m ("100 millicores") is a tenth of a core.
• Memory is measured in bytes, usually written as Mi (mebibytes, about a million bytes) or Gi (gibibytes, about a billion).

What happens over the limit

CPU and memory behave very differently when a container hits its limit:

• CPU: it gets slowed down. CPU can be shared moment to moment, so a container that wants more than its CPU limit is simply throttled. It keeps running, just slower. Nothing crashes, which also makes it easy to miss.
• Memory: it gets killed. Memory can't be taken back from a running program, so a container that goes over its memory limit is stopped immediately: OOMKilled, the failure you met in Lesson 3. The kubelet restarts it, and if it keeps happening, it ends up in CrashLoopBackOff.

Requests also protect your pod. If a node itself runs low on memory, Kubernetes removes pods to save it, starting with those that set no requests at all.

Try it: see your reservations

Start the three-node cluster from Lesson 2 (you'll need its extra nodes at the end of this lesson), and give kubectl a way to measure usage. kind doesn't include metrics-server, the component behind kubectl top, so install it, with one setting that kind needs:

kind create cluster --name learn --config kind-learn.yaml
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl patch deployment metrics-server -n kube-system --type=json -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

What each command does:

You don't need to understand the third command in detail. It switches off a certificate check that kind's practice nodes can't pass. That's fine on a practice cluster, and not something you'd do on a real one.

Now save the Deployment above (with its resources section) as deploy.yaml, using replicas: 3, and apply it. Then look at the node's books:

kubectl apply -f deploy.yaml
kubectl describe node learn-worker

Scroll to Allocated resources. It shows the total requested and limited for CPU and memory, as a percentage of what the node can give. That's the scheduler's view: reservations, not real use. For real use, wait a minute for metrics to arrive, then:

kubectl top pods
kubectl top node

The two views rarely match, and the gap between them is where most resource problems hide.

Break it on purpose

1. Ask for more than any node has. Change the request to memory: 100Gi, apply, and look:

kubectl get pods
kubectl describe pod <one-of-the-pending-pods>

New pods sit in Pending, and the events say something like 0/1 nodes are available: 1 Insufficient memory. Nothing is broken; there's just nowhere with that much unreserved memory. Meanwhile your old pods keep running, because the rolling update from Lesson 5 waits for new pods to be ready. Put the request back to 64Mi and apply.

2. Go over a memory limit. This pod, from the official Kubernetes docs, tries to use 250 MB with a limit of 100Mi. Save it as memory-hog.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: memory-hog
spec:
  containers:
    - name: hog
      image: polinux/stress
      resources:
        requests:
          memory: 50Mi
        limits:
          memory: 100Mi
      command: ["stress"]
      args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "1"]

kubectl apply -f memory-hog.yaml
kubectl get pod memory-hog --watch

Within seconds it shows OOMKilled, then CrashLoopBackOff with a climbing restart count. Press Ctrl+C, and confirm the reason:

kubectl describe pod memory-hog | grep -A3 "Last State"
kubectl delete pod memory-hog

Picking the numbers

Guessing is how most clusters end up either wasteful or fragile. A simple approach:

• Measure first. Run the app under normal load and watch kubectl top pods for a while, ideally over days, not minutes.
• Request what it typically uses, so the scheduler's books match reality.
• Set the memory limit with headroom above its normal peak, so a busy moment doesn't get it killed.
• CPU limits are optional. Going over only slows a container down, so many teams set a CPU request but no CPU limit. Either way is fine; just know which you chose.

Choosing nodes

Requests decide whether a pod fits on a node. Sometimes you also care which node. In Lesson 2 you used the main tool, nodeSelector, to send a pod to nodes with a label. There are three more ways, from bluntest to most flexible.

nodeName: this exact node. It skips the scheduler entirely:

spec:
  nodeName: learn-worker2

It's handy for debugging, but if that node is full or gone, the pod simply fails, with no scheduler to find it another home. Avoid it in real workloads.

Node affinity: a more flexible nodeSelector. It lets you write "must" and "prefer" rules, like "must have an SSD, and preferably be in zone A". You'll meet it in real clusters; for now, nodeSelector covers what you need.

Taints and tolerations: keeping pods off nodes. The other tools pull pods toward nodes. A taint pushes them away: it's a "keep out" sign on a node, and only pods with a matching toleration may be scheduled there. You've seen one already: the control-plane node's taint, from Lesson 2. Try your own:

kubectl taint nodes learn-worker2 dedicated=gpu:NoSchedule

New pods now avoid learn-worker2. A pod that should be allowed there carries a toleration:

spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: gpu
      effect: NoSchedule

A toleration only permits a pod onto the node; it doesn't send it there. To reserve a node for certain pods, use both: a taint to keep others out, and a selector or affinity to pull the right ones in. Remove the taint with the same command plus a minus sign at the end: kubectl taint nodes learn-worker2 dedicated=gpu:NoSchedule-.

Clean up

kind delete cluster --name learn

Check yourself

Key terms

• Request: the CPU and memory a container is guaranteed, and what the scheduler uses to place it.
• Limit: the most a container may use. Over the CPU limit it's throttled; over the memory limit it's OOMKilled.
• Allocatable: how much of a node's CPU and memory pods can reserve, after the system takes its share.
• Node affinity: a more flexible nodeSelector, with "must" and "prefer" rules.
• Taint and toleration: a "keep out" sign on a node, and the permission a pod needs to ignore it.

Go deeper

• Resource management for pods and containers (opens in a new tab) in the official docs.
• Assign memory resources (opens in a new tab) is where the memory-hog example comes from.
• Assigning pods to nodes (opens in a new tab) and Taints and tolerations (opens in a new tab) cover every scheduling option.