Load balancers and reverse proxies

Where you are

In Lesson 1, api.civicgrid.org answered with server: cloudflare, and kwamib.dev with server: Vercel. Neither is the program that actually holds the data. They're something in front of it, passing requests along. Lesson 9's callout said the same about TLS: Cloudflare ends your encrypted connection and opens its own to the servers behind.

This lesson is about those things in the middle. They're in front of almost every real website and API, and they're where many of the errors you'll meet come from.

A host at the door

A busy restaurant doesn't let diners wander in and pick a waiter. A host at the door greets everyone and seats them, spreading the tables across the waiters so nobody's swamped. If a waiter goes on a break, the host stops sending people to their section. And diners only ever deal with the host at the door; they never need to know how many waiters are working tonight.

That host is two ideas at once:

• A reverse proxy is something that receives requests on behalf of servers behind it, and passes them on. Visitors talk to the proxy; the proxy talks to the servers.
• A load balancer is a reverse proxy with several servers behind it, which spreads the requests between them and stops sending to any that aren't well.

"Reverse" is there to tell it apart from a forward proxy, which works for the visitor instead. A company might send all its staff's web browsing through a forward proxy, to filter or log it. A forward proxy speaks for the people asking; a reverse proxy speaks for the servers answering.

Why put something in the middle

It sounds like an extra hop for nothing. It's one of the most useful hops on the internet:

• More servers than one. One server can only handle so much. With a load balancer in front, you can run five copies of your app and add a sixth on a busy day, without visitors noticing.
• Surviving failures. If a server crashes, the load balancer stops using it, and visitors carry on with the others. It's the copies in two zones from Lesson 7, put to work.
• One public door. The servers can stay in private subnets, reachable only from the load balancer, exactly like the security-group chain in Lesson 8.
• Doing the shared work once. TLS, compressing responses, blocking attacks, caching popular pages: the proxy can do these for every server behind it.

Build a load balancer

The fastest way to understand one is to build one. You'll run two tiny servers and put a load balancer of your own in front of them, all on your computer. If Lesson 4's server is still running on port 8000, stop it first with Ctrl+C.

In a new folder, make two "servers", each with its own page:

mkdir lb-test
cd lb-test
mkdir a b
echo 'Hello from server A' > a/index.html
echo 'Hello from server B' > b/index.html

Start each one in its own terminal, from inside lb-test. --directory tells Python's server which folder to serve, and --bind 127.0.0.1 keeps it listening only on your computer, from Lesson 4:

python3 -m http.server 8001 --bind 127.0.0.1 --directory a

python3 -m http.server 8002 --bind 127.0.0.1 --directory b

Now the load balancer. In a third terminal, create proxy.py in lb-test:

from http.server import BaseHTTPRequestHandler, HTTPServer
import urllib.error
import urllib.request

backends = ["http://127.0.0.1:8001", "http://127.0.0.1:8002"]
turn = 0


class Proxy(BaseHTTPRequestHandler):
    def do_GET(self):
        global turn
        backend = backends[turn % len(backends)]
        turn = turn + 1
        print("sending", self.path, "to", backend)

        try:
            with urllib.request.urlopen(backend + self.path, timeout=3) as reply:
                self.answer(reply.status, reply.read())
        except urllib.error.HTTPError as error:
            self.answer(error.code, error.read())
        except OSError:
            self.answer(502, b"502 Bad Gateway: the server behind me didn't answer\n")

    def answer(self, status, body):
        self.send_response(status)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


HTTPServer(("127.0.0.1", 8000), Proxy).serve_forever()

It's short, but it's a real reverse proxy. Piece by piece:

• backends is the list of servers behind it. People often call them backends, or upstreams.
• Proxy is a class, the kind of blueprint from Python Lesson 2. Python's http.server calls its do_GET for every GET request that arrives, and answer is a small helper that sends a reply back.
• turn % len(backends) picks the next server in turn: % is the remainder after dividing, so with two backends it goes 0, 1, 0, 1. That's round robin, the simplest way to spread requests. global turn lets the function change the counter that lives outside it.
• urllib.request.urlopen makes the request to the chosen backend, the same job the requests library did in Python Lesson 6, but built into Python, so there's nothing to install.
• The except blocks, from Python Lesson 4: if the backend answers with an error like a 404, pass it on as it is; if it doesn't answer at all, reply with a 502. More on that below.

Run it:

python3 proxy.py

Then, in a fourth terminal, send it a few requests:

curl -s http://127.0.0.1:8000/
curl -s http://127.0.0.1:8000/
curl -s http://127.0.0.1:8000/
curl -s http://127.0.0.1:8000/

Hello from server A
Hello from server B
Hello from server A
Hello from server B

Same address, same port, every time, and yet the answers alternate. curl has no idea there are two servers. The proxy's terminal shows what it did:

sending / to http://127.0.0.1:8001
sending / to http://127.0.0.1:8002
sending / to http://127.0.0.1:8001
sending / to http://127.0.0.1:8002

Interleaved with these, you'll see the proxy's own request log lines, one per request. That's the whole idea of a load balancer: one door, many servers, the visitor never the wiser.

Layer 4 and layer 7

Load balancers come in two broad kinds, named after layers in a standard model of networking that you'll see mentioned everywhere: layer 4 is TCP and UDP from Lesson 4, and layer 7 is HTTP.

• A layer 4 load balancer works with connections. It sees addresses and ports, picks a server, and passes the bytes through without reading them. It's fast, and works for anything over TCP or UDP, like databases.
• A layer 7 load balancer reads each HTTP request, like yours does with self.path. So it can make choices by what's asked: send /api/ to the API servers and everything else to the website, or route by the Host header from Lesson 4, so one load balancer serves many sites.

AWS offers both: a Network Load Balancer works at layer 4, and an Application Load Balancer at layer 7. Cloudflare and Vercel are layer 7: they read every request, which is how they know which site you asked for.

Health checks

Your load balancer has a weakness, and you'll find it in the first case below: it keeps sending requests to a server that's gone. Real load balancers avoid that with health checks. Every few seconds, they ask each server a simple question, often a request to a path like /health, and if a server fails a few in a row, they stop sending it traffic until it passes again. Kubernetes does the same inside a cluster with its readiness checks, which the Kubernetes track's health checks lesson covers.

TLS termination and forwarded headers

A layer 7 load balancer can only read requests it can decrypt, so for HTTPS it has to be where the TLS connection ends. That's TLS termination, from Lesson 9's callout: the certificate lives on the load balancer, and the servers behind it don't need one each. What happens after depends on the setup: the proxy can talk to the servers over plain HTTP, inside a private network, or start a fresh TLS connection to them, which is safer.

There's a side effect. To the servers, every request now comes from the proxy, not the visitor. Look at the log in server A's terminal:

127.0.0.1 - - [24/Oct/2026 23:46:02] "GET / HTTP/1.1" 200 -

That 127.0.0.1 is your proxy, not curl. On a real site, every request would look as if it came from the load balancer's address. So proxies add forwarded headers to each request as they pass it on:

• X-Forwarded-For carries the visitor's real address, for logs, rate limits and security rules.
• X-Forwarded-Proto says whether the visitor used https or http, since the server can no longer see it for itself.

Your proxy doesn't add them; real ones do. And there's a catch worth remembering: anyone can put an X-Forwarded-For header in their own request, so a server should only trust it when it came from its own proxy.

Break it on purpose

Three cases, two of them with your own load balancer. Work out the cause before you read the explanation.

One server down: 502. Keep everything running, then stop server B with Ctrl+C in its terminal. Send four requests again, this time printing the status code after each:

curl -s -w '%{http_code}\n' http://127.0.0.1:8000/

Run it four times:

Hello from server A
200
502 Bad Gateway: the server behind me didn't answer
502
Hello from server A
200
502 Bad Gateway: the server behind me didn't answer
502

Half the requests fail. The proxy is fine and answering, but when it's B's turn, B refuses the connection, and the proxy has nothing to pass on. 502 Bad Gateway means exactly that: "I'm working, but the server behind me failed". It's the most common error you'll see from a proxy, and it tells you where to look: not at the proxy, but at the servers behind it. Are they running? On the port the proxy expects? That's Lesson 4's refused connection, one hop further back.

Teach it to step around. Real load balancers don't fail half the time when one server is down, so make yours smarter. Stop proxy.py with Ctrl+C, and create proxy2.py next to it. Only do_GET changes: it now tries each backend in turn until one answers.

from http.server import BaseHTTPRequestHandler, HTTPServer
import urllib.error
import urllib.request

backends = ["http://127.0.0.1:8001", "http://127.0.0.1:8002"]
turn = 0


class Proxy(BaseHTTPRequestHandler):
    def do_GET(self):
        global turn
        for attempt in range(len(backends)):
            backend = backends[turn % len(backends)]
            turn = turn + 1
            print("sending", self.path, "to", backend)

            try:
                with urllib.request.urlopen(backend + self.path, timeout=3) as reply:
                    self.answer(reply.status, reply.read())
                    return
            except urllib.error.HTTPError as error:
                self.answer(error.code, error.read())
                return
            except OSError:
                print(backend, "didn't answer, trying the next one")

        self.answer(503, b"503 Service Unavailable: no server behind me is answering\n")

    def answer(self, status, body):
        self.send_response(status)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


HTTPServer(("127.0.0.1", 8000), Proxy).serve_forever()

Run python3 proxy2.py, and send the same requests, with B still stopped. Every one now comes back 200, from server A, and the proxy's terminal shows why:

sending / to http://127.0.0.1:8002
http://127.0.0.1:8002 didn't answer, trying the next one
sending / to http://127.0.0.1:8001

It tried B, got no answer, and moved on to A. That's a retry, which is the quick version of what health checks do ahead of time. Now stop server A too, and send one more:

503 Service Unavailable: no server behind me is answering
503

With nothing left to try, the answer changes: 503 Service Unavailable means "there's no healthy server to send you to at all". On a real site, a 503 usually means every copy of the app is down, or still starting up, or failing its health checks. When you're done, stop proxy2.py with Ctrl+C.

Slow, not broken: 504. Imagine one of the API's requests runs a heavy database query that takes two minutes. The visitor waits, then gets a 504 Gateway Timeout. The server never crashed and was still working on the answer, but every proxy has a limit on how long it will wait, like the timeout=3 in your script, and it gave up first. A 504 points at something slow behind the proxy: a stuck query, an overloaded server, or a call to another service that never comes back. The fix is to make the slow work faster, or move it out of the request, not to raise the timeout until it goes away.

Put together, the three gateway errors tell you where to look:

Where you are now

You've built a working load balancer, made it step around a broken server, and you can read 502, 503 and 504 like signposts. You know where TLS ends, why servers see the proxy's address, and the difference between layer 4 and layer 7. Every piece of this track now has a home: addresses, ranges, ports, DNS, routing, VPCs, firewalls, TLS and proxies. Lesson 11 puts them all inside one Kubernetes cluster, where pods, Services and gateways do the same jobs at smaller scale.

Check yourself

Key terms

• Reverse proxy: something that receives requests on behalf of servers behind it, and passes them on.
• Forward proxy: a proxy that works for the visitor, like a company's web filter.
• Load balancer: a reverse proxy that spreads requests across several servers.
• Backend (upstream): a server behind a proxy.
• Round robin: taking each server in turn.
• Health check: a regular test of each server, so broken ones stop getting traffic.
• Layer 4 and layer 7: balancing connections by address and port, or balancing HTTP requests by what they ask for.
• TLS termination: ending the encrypted connection at the proxy instead of the server.
• X-Forwarded-For: the header that carries the visitor's real address past a proxy.

Go deeper

• Cloudflare's Learning Center on what a reverse proxy is (opens in a new tab).
• MDN's pages on 502 (opens in a new tab), 503 (opens in a new tab) and 504 (opens in a new tab).