Putting it together: networking in Kubernetes

By the end you'll know

The three address ranges every cluster has, and why they mustn't overlap
How pods on different nodes reach each other without NAT
Why a Service is a load balancer with an address no machine owns
How cluster DNS works, and what the search list and ndots do
How to write a NetworkPolicy, Kubernetes's firewall between pods

Where you are

You now know every piece a network is made of: addresses and octets (Lesson 2), ranges (Lesson 3), ports (Lesson 4), DNS (Lesson 5), routing (Lesson 6), private networks (Lesson 7), firewalls (Lesson 8), TLS (Lesson 9) and load balancers (Lesson 10).

A Kubernetes cluster uses every one of them, packed into a few machines. This lesson isn't new theory. It's the moment the whole track clicks together, by watching each idea at work inside a real cluster. If you've done the Kubernetes track, you've already used Services and the Gateway API there. This lesson shows what's underneath them.

A campus with its own phone system

Picture a big company campus. Every desk has its own extension, but people move desks all the time, so their extensions keep changing. So each department also has a fixed department number: you call "Accounts", and it rings whoever's in Accounts today. There's an internal directory, so nobody has to remember numbers. And some doors need a badge: not every department lets everyone in.

A cluster is that campus:

Pods are the people. Each one gets its own address, and it changes every time a pod is replaced.
Services are the department numbers: one fixed address that reaches whichever pods are doing that job right now.
Cluster DNS is the internal directory: pods ask for web, not 10.96.208.160.
NetworkPolicy is the badge door: rules for which pods may talk to which.

Build a cluster to look inside

You'll need kind and kubectl, from the Kubernetes track's first cluster lesson, and Docker running. Make a config for a cluster with one control plane and two workers:

BASH

cat > kind-net.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF

cat > kind-net.yaml <<'EOF' writes everything up to the line EOF into the file, a handy way to create a small file from the terminal. Then create the cluster:

BASH

kind create cluster --name net --config kind-net.yaml

Among its lines you'll see Installing CNI. Hold that thought: it's the networking, and it comes up below.

Three ranges, never overlapping

A cluster needs addresses for three different kinds of thing, and each kind gets its own range. Start with the machines, the nodes:

BASH

kubectl get nodes -o wide

OUTPUT

NAME                STATUS     ROLES           AGE   VERSION   INTERNAL-IP
net-control-plane   NotReady   control-plane   10s   v1.35.0   172.18.0.3
net-worker          NotReady   <none>          1s    v1.35.0   172.18.0.4
net-worker2         NotReady   <none>          1s    v1.35.0   172.18.0.2

A few columns are trimmed. NotReady just means the cluster is seconds old: a node turns Ready once its networking has started, so run it again shortly and you'll see Ready. In kind, each node is a Docker container, and 172.18.0.x is the private network Docker made for them: part of the 172.16.0.0/12 range from Lesson 3. On a cloud, these would be addresses from your VPC's subnets, as in Lesson 7. Next, each node's share of the pod range:

BASH

kubectl get nodes -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.podCIDR

OUTPUT

NAME                PODCIDR
net-control-plane   10.244.0.0/24
net-worker          10.244.2.0/24
net-worker2         10.244.1.0/24

The cluster's pod range is 10.244.0.0/16, and each node was handed one /24 of it, exactly the split from Lesson 3: 256 addresses per node, for that node's pods. And the third range, for Services:

BASH

kubectl cluster-info dump | grep -m1 -o "service-cluster-ip-range=[^\"]*"

OUTPUT

service-cluster-ip-range=10.96.0.0/16

cluster-info dump prints a lot of the cluster's settings, and grep -m1 -o picks out just the first match. Every Service will get an address from 10.96.0.0/16. So, in one table:

What	Range	Who hands them out
Nodes	`172.18.0.x` here	Docker here; your VPC on a cloud
Pods	`10.244.0.0/16`, a `/24` per node	The cluster, through its network add-on
Services	`10.96.0.0/16`	The cluster's control plane

Try the pod range in the slider from Lesson 3. At /16 it's the whole cluster's pod range; drag to /24 to see one node's share:

The three ranges must never overlap with each other, and, the one that catches people, they must not overlap with anything the cluster needs to reach either: your VPC, your office network, a database in another network. It's Lesson 3's VPN problem and Lesson 7's peering problem, one more time. On a managed cloud cluster, you usually pick the pod and Service ranges when you create the cluster, and you can't change them afterwards.

Pods reach each other directly

Run a small web app with three copies, and look at where the pods landed:

BASH

kubectl create deployment web --image=nginx --replicas=3
kubectl rollout status deployment/web --timeout=120s
kubectl get pods -o wide

OUTPUT

NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE
web-68d995574f-sdrpd   1/1     Running   0          16s   10.244.2.2   net-worker
web-68d995574f-w6rg7   1/1     Running   0          16s   10.244.2.4   net-worker
web-68d995574f-xjfzd   1/1     Running   0          16s   10.244.2.3   net-worker

Columns are trimmed again. Each pod has its own address, and all three come from 10.244.2.0/24, because all three landed on net-worker, the node that owns that slice. Yours may spread across both workers; the scheduler decides.

Kubernetes has one big rule for its network: every pod can reach every other pod by its address, directly, without NAT, whichever node it's on. A pod on net-worker2 can reach 10.244.2.2 on net-worker as if they were in the same room.

That's the job of the CNI you saw being installed: the Container Network Interface, a plug-in that gives pods their addresses and connects them. Different clusters use different ones. kind's own is called kindnet, and many clouds use Cilium or Calico. A simple CNI like kindnet does it with exactly what you learned in Lesson 6: it adds a route on every node for every other node's pod range. In plain words, net-worker2's routing table gets rules like these:

TEXT

10.244.0.0/24 via 172.18.0.3    the control plane's pods
10.244.2.0/24 via 172.18.0.4    net-worker's pods

A packet for 10.244.2.2 matches the second rule, goes to net-worker's node address, and net-worker hands it to the pod. Routing tables, longest match, next hop: nothing new, just inside a cluster.

The kind cluster from this lesson. Each node owns a slice of the pod range, routes join the slices up, and Services sit above it all on addresses of their own.

A Service is a load balancer with a made-up address

Pod addresses change every time a pod is replaced, so nothing should call them directly. That's what Services are for, as the Kubernetes track's Services lesson showed. Put one in front of web:

BASH

kubectl expose deployment web --port=80
kubectl get svc web

OUTPUT

NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
web    ClusterIP   10.96.208.160   <none>        80/TCP    0s

10.96.208.160, from the Service range. Now ask who's behind it:

BASH

kubectl get endpointslices -l kubernetes.io/service-name=web

OUTPUT

NAME        ADDRESSTYPE   PORTS   ENDPOINTS                          AGE
web-b2nrt   IPv4          80      10.244.2.4,10.244.2.3,10.244.2.2   0s

The three pods' addresses. That's Lesson 10's backends list, kept up to date by the cluster as pods come and go. A Service is a load balancer: one address in front, a list of backends behind.

But here's the strange part: no machine has the address 10.96.208.160. There's no network card with that address on any node. Instead, a component called kube-proxy, or the CNI itself on some clusters, writes rules into every node that say: "anything sent to 10.96.208.160:80, quietly rewrite the destination to one of those three pods". The rewriting happens on whichever node the request starts from, before it ever leaves. It's a load balancer with no box: just rules, everywhere at once. Which is why ping to a Service address often gets no answer, even when the Service works perfectly: there's nothing there to reply.

Finding things by name: cluster DNS

The cluster runs its own DNS server, just like the resolver from Lesson 5, as a Service of its own:

BASH

kubectl get svc -n kube-system kube-dns

OUTPUT

NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   24s

53/UDP, from Lesson 4's port table: DNS, mostly over UDP. Now look from inside a pod. This starts a throwaway pod with BusyBox, a tiny Linux image with basic tools, runs three commands in it, and deletes it afterwards:

BASH

kubectl run tmp --rm -i --restart=Never --image=busybox:1.36 -- sh -c 'cat /etc/resolv.conf; echo ---; nslookup web; echo ---; wget -qO- -T 3 http://web | head -4'

The first part is the pod's DNS settings:

OUTPUT

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

nameserver 10.96.0.10 is the cluster's DNS Service, so every lookup from this pod goes there, the way your Mac sent its lookups to your router in Lesson 5.
search is a list of endings to try when a name is short. Ask for web, and the pod tries web.default.svc.cluster.local, then web.svc.cluster.local, then web.cluster.local.
ndots:5 says when to use that list: any name with fewer than 5 dots is tried with the endings first.

The next part shows the search list at work:

OUTPUT

Server:		10.96.0.10
Address:	10.96.0.10:53

** server can't find web.cluster.local: NXDOMAIN

** server can't find web.svc.cluster.local: NXDOMAIN

Name:	web.default.svc.cluster.local
Address: 10.96.208.160

Some duplicate lines are trimmed. BusyBox tried the endings, got Lesson 5's NXDOMAIN for the ones that don't exist, and found web.default.svc.cluster.local, at the Service's address. That full name is the pattern for every Service: <service>.<namespace>.svc.cluster.local. Because this pod is in the default namespace, plain web was enough. And the last part fetched the page through it all:

OUTPUT

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

Name, to DNS, to a Service address, to a pod on another node, all in one request: Lessons 5, 10 and 6, in a row.

The search list has a cost worth knowing about. With ndots:5, a name like api.civicgrid.org has only 2 dots, so a pod tries api.civicgrid.org.default.svc.cluster.local and the other endings first, and gets an NXDOMAIN for each, before trying the real name. It works, but every lookup of an outside name costs several extra questions. Busy apps sometimes add a final dot, api.civicgrid.org., which means "this name is complete", from Lesson 5's invisible root dot, so the search list is skipped.

NetworkPolicy: a firewall between pods

By default, every pod can reach every other pod, in any namespace. That's convenient, and it's the same as a VPC with no security groups: one broken or hacked pod can talk to everything, including your database. NetworkPolicy is Kubernetes's answer, and it works like Lesson 8's firewalls, with labels instead of addresses.

Here's one that says "only pods labelled role: frontend may reach the web pods":

BASH

cat > deny-web.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-only-from-frontend
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
EOF
kubectl apply -f deny-web.yaml

Read it from the top:

podSelector picks which pods this policy protects: the ones labelled app: web, which kubectl create deployment web gave them.
policyTypes: Ingress means it controls traffic coming in, from Lesson 8.
ingress.from is the allow list: pods labelled role: frontend.

The moment a policy selects a pod, that pod switches to default deny for the direction the policy covers: anything not on the allow list is blocked. Just like a security group, which has no deny rules, only allow rules. And like a security group using another group as a source, it allows "pods with this label", wherever they are and whatever their address.

Test it with two throwaway pods, one without the label and one with it. Give it a few seconds to take effect first:

BASH

kubectl run tmp2 --rm -i --restart=Never --image=busybox:1.36 -- sh -c 'wget -qO- -T 3 http://web | head -2 || echo FAILED'

OUTPUT

wget: download timed out

BASH

kubectl run tmp3 --rm -i --restart=Never --image=busybox:1.36 --labels=role=frontend -- sh -c 'wget -qO- -T 3 http://web | head -2 || echo FAILED'

OUTPUT

<!DOCTYPE html>
<html>

You may also see If you don't see a command prompt, try pressing enter., which you can ignore. The unlabelled pod's request timed out, and the frontend one got the page. Notice it timed out, not refused: like a cloud firewall, the policy drops traffic silently. Lesson 8's rule of thumb holds here too: a timeout between pods that worked yesterday often means a policy.

There's one catch: NetworkPolicy only works if the CNI enforces it. The cluster accepts your policy either way, but a CNI that doesn't support policies simply ignores it, and everything stays open, with no warning. Recent versions of kind's kindnet enforce it, which is why the test above worked; Cilium and Calico do too. Always test a policy the way you just did, rather than trusting it.

In my real setup

Everything in this lesson is in CivicGrid's cluster. The nodes have private addresses in a DigitalOcean VPC (Lesson 7). Pods get addresses from the cluster's own pod range, and the API reaches its Redis cache by its full Service name, civicgrid-redis.civicgrid.svc.cluster.local. Nothing has a public address: requests arrive through a Cloudflare Tunnel, a connection a pod inside the cluster opens outward, then pass through Envoy Gateway to the API's ClusterIP Service. The Kubernetes track's Services and Gateway API lessons show those last steps.

Clean up

Delete the cluster and the two files:

BASH

kind delete cluster --name net
rm kind-net.yaml deny-web.yaml

Break it on purpose

Three cases. The first one you've already seen; work out the cause of the others before you read the explanation.

A policy nobody remembers. You saw it above: a pod that could reach web a minute ago now times out, and nothing changed in either app. Someone added a NetworkPolicy. When pods can't talk and the error is a timeout, list the policies in the namespace with kubectl get networkpolicy, and read which pods each one selects and allows. A refused connection would point elsewhere: the Service or the pods, as Lesson 4 taught.

The right name, the wrong namespace. An app in a namespace called frontend calls http://web, and gets Could not resolve host, or bad address from BusyBox. The Service is running fine, in default. Why? A pod's search list starts with its own namespace: from frontend, plain web becomes web.frontend.svc.cluster.local, which doesn't exist, so every ending fails with NXDOMAIN. Across namespaces, use web.default, or the full web.default.svc.cluster.local. It's Lesson 5's NXDOMAIN, and the fix is always to check the full name.

A cluster that can't reach the database. Your company's database lives in another network at 10.244.1.50. A new cluster's pods can reach everything else, but connections to the database go nowhere. The database is up, and machines outside the cluster reach it fine. What's going on? Look at the pod range: 10.244.0.0/16. The database's address falls inside it, in the slice 10.244.1.0/24 that belongs to net-worker2. Every node has a route saying "10.244.1.0/24 is over on net-worker2", so the packet goes there, the most specific match from Lesson 6, instead of out to the real database. The ranges overlap, Lesson 3's lesson for the last time. Because a cluster's ranges can't usually be changed after it's created, the real fix is rebuilding the cluster with ranges that clash with nothing, which is why they're worth planning before you build, alongside the VPC.

Where you are now

You can find a cluster's three ranges, explain how pods reach each other across nodes, see through a Service to the pods behind it, read a pod's DNS settings, and lock down traffic between pods with a NetworkPolicy. Every lesson in this track showed up inside one cluster. What's left is the skill that ties it all together when something breaks: Lesson 12 turns the whole track into a troubleshooting checklist, from "is the name right?" to "is a policy dropping it?", in the order that finds problems fastest.

Check yourself

Why must a cluster's pod and Service ranges not overlap with the VPC?
Each node got a /24 from 10.244.0.0/16. How many pod addresses does each node have?
Why does ping to a Service's ClusterIP often get no reply?
A pod in the frontend namespace can't resolve web. The Service is in default. What's the fix?
A NetworkPolicy selects the web pods and allows only role: frontend. What happens to other pods?
You apply a NetworkPolicy, but nothing is blocked. What's a likely cause?

Key terms

Node range, pod range, Service range: the three address ranges in every cluster. None may overlap with each other, or with networks the cluster needs to reach.
CNI (Container Network Interface): the plug-in that gives pods their addresses and connects them, like kindnet, Cilium or Calico.
ClusterIP: a Service's address. No machine owns it; rules on every node send its traffic to a pod.
kube-proxy: the component that writes those rules on each node, on many clusters.
EndpointSlice: the cluster's up-to-date list of the pods behind a Service.
Cluster DNS: the cluster's own DNS server. Full names look like web.default.svc.cluster.local.
Search list and ndots: the endings a pod tries for short names, and the dot count below which it tries them.
NetworkPolicy: rules for which pods may talk to which, using labels. Enforced only by CNIs that support it.

Go deeper

The Kubernetes documentation on Network Policies (opens in a new tab), with more examples.
The Kubernetes documentation on DNS for Services and Pods (opens in a new tab).