Ports and protocols: TCP, UDP and HTTP

Where you are

Lessons 2 and 3 were about addresses: how data finds the right machine. But one machine runs many programs at once. A server might run a website, a database and a remote login service, all on the same address. When data arrives, which program is it for?

You've already seen the answer twice without it being explained. In Lesson 1, curl said Trying 104.21.69.165:443. In Lesson 2, it said Failed to connect to 127.0.0.1 port 8000. That number after the colon is a port, and this lesson is about ports and the rules programs follow when they talk through them.

Ports: one address, many doors

A company has one phone number, and each person has an extension. The main number gets your call to the building; the extension gets it to the right desk. An IP address and a port work the same way: the address gets data to the machine, and the port gets it to the right program on that machine.

A port is just a number, from 0 to 65,535. That odd limit is the same binary you've been using: a port is 16 bits, and 16 bits make 65,536 values, exactly like the /16 in Lesson 3. Written together, an address and a port look like 104.21.69.165:443.

A program that's waiting for connections is said to listen on a port. When a request arrives for that port, the operating system hands it to the program listening there. If nothing is listening, the machine says no, and that's the Connection refused from Lesson 2.

Some ports are used so widely that they're conventions everyone follows. You don't need to memorise them, but you'll start recognising them:

When you type https://api.civicgrid.org into a browser, you never type 443, because https:// means 443 unless you say otherwise. That's why curl in Lesson 1 went straight to port 443.

Your side of the connection has a port too

Every connection has two ends, and both have an address and a port. The server listens on a fixed, well-known port, so you know where to find it. Your side gets a temporary port, picked at random by your operating system for that one connection. Look back at Lesson 2's error:

* connect to 127.0.0.1 port 8000 from 127.0.0.1 port 33068 failed: Connection refused

Port 8000 was the door curl knocked on; port 33068 was the temporary one curl was knocking from. Run it again and you'll get a different number. The four together, both addresses and both ports, identify one connection, which is how a server can talk to thousands of visitors on port 443 at the same time without mixing them up.

Run a server and knock on its door

Time to make a program listen. Python comes with a tiny web server built in. Make a new folder, put one web page in it, and start the server on port 8000:

mkdir port-test
cd port-test
echo '<h1>Hello from port 8000</h1>' > index.html
python3 -m http.server 8000

echo prints text, and > sends it into a file instead of the screen, so the third line creates index.html. python3 -m http.server 8000 runs Python's web server module, serving the files in this folder on port 8000. You'll see:

Serving HTTP on :: port 8000 (http://[::]:8000/) ...

The exact wording varies between Python versions and computers, but it's telling you the server is listening. It keeps running, so leave that terminal open and open a second one. In the second terminal, ask your computer what's listening:

lsof -nP -iTCP:8000 -sTCP:LISTEN

lsof lists open files, and on Mac and Linux network connections count as files. The options narrow it to TCP connections on port 8000 that are listening, and show numbers instead of names. You'll see something like:

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
python3 51234 you     4u  IPv6 0x1234      0t0  TCP *:8000 (LISTEN)

That's your server: the program python3, listening on port 8000. The * means it's listening on every address the computer has. Now knock on the door:

curl -v http://127.0.0.1:8000/

The important lines:

*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> GET / HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.7.1
> Accept: */*
>
< HTTP/1.0 200 OK
< Server: SimpleHTTP/0.6 Python/3.12.4
< Content-type: text/html
< Content-Length: 30
<
<h1>Hello from port 8000</h1>

Your version numbers will differ. Lines starting with > are what curl sent, lines starting with < are what came back, and the last line is your page. Look at your first terminal too: the server printed a line for your request. Now stop the server there with Ctrl+C, and run the same curl again. It's Connection refused: same machine, same port, but nobody listening.

The lines between Trying and Connected hide two rules working together, TCP and HTTP. The next two sections open them up.

TCP: a phone call

A phone call starts with a ritual: you dial, they pick up and say hello, you say hello back, and only then do you start talking. If a word gets lost, you say "sorry, say that again". When you're done, you both say goodbye.

TCP (Transmission Control Protocol) is that phone call for computers. Before any data moves, the two sides do a three-step handshake: one says "can we talk?", the other says "yes, can you hear me?", and the first says "yes". curl printed Connected once that handshake finished. After that, TCP numbers every piece of data, the other side confirms what it received, and anything lost is sent again. Everything arrives, and in the right order.

That reliability is why websites, APIs, SSH and databases all use TCP: a web page with a missing chunk, or a database write that half arrived, would be useless. The cost is a little time, for the handshake and the confirmations.

UDP: a postcard

A postcard has no ritual. You write it, address it, drop it in the box, and hope. No one confirms it arrived, and if two are sent, they might arrive in either order.

UDP (User Datagram Protocol) is the postcard. There's no handshake and no confirmation; each message is sent on its own, and if it's lost, it's lost. That sounds worse, but it's faster, and for some jobs speed matters more than perfection:

• DNS lookups are usually UDP: one short question, one short answer. If the answer doesn't come, just ask again. You'll see this in Lesson 5.
• Video calls and online games use UDP, because a late piece of video is no use. It's better to skip it and show the next one than to pause everyone while it's resent.

Both TCP and UDP use ports, but they're separate sets: TCP port 53 and UDP port 53 are different doors.

HTTP: the conversation itself

TCP gets a reliable line open. It doesn't say what to say on it. That's the job of a higher-level protocol, a set of rules for a particular kind of conversation, and for the web that protocol is HTTP.

An HTTP request is just text. The one curl sent was:

GET / HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: curl/8.7.1
Accept: */*

The first line is the important one: a method, GET, meaning "give me"; a path, /, meaning the home page; and the HTTP version. The lines after it are headers: extra details, like which site you want and what program is asking. Host matters more than it looks: one server can host many sites, and that's how it knows which one you meant. Other methods you'll meet include POST, for sending data, like a form, and DELETE, for removing something.

The response is text too. Its first line holds a status code, a three-digit number that tells you how it went, and after the headers comes the body: the page itself. The status codes you'll see most:

A rough rule: codes in the 200s mean it worked, 300s mean go elsewhere, 400s mean the problem is on the asking side, and 500s mean it's on the server side.

The 200 came back as HTTP/1.0 here because Python's simple server only speaks the oldest version. In Lesson 1, Cloudflare answered with HTTP/2, a newer, faster version with the same methods, headers and status codes underneath. And HTTPS is HTTP sent inside an encrypted connection, which is Lesson 9.

Talk HTTP by hand

Because HTTP is only text, you don't need curl at all. Start the server again in your first terminal, from inside port-test:

python3 -m http.server 8000

Then in the second, use nc (netcat), a tool that opens a plain TCP connection and sends whatever you give it:

printf 'GET / HTTP/1.0\r\n\r\n' | nc 127.0.0.1 8000

printf prints text exactly as written, where \r\n is the line ending HTTP expects, and the blank line at the end means "that's my whole request". The | passes that text to nc, which sends it to port 8000. Back comes:

HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.12.4
Content-type: text/html
Content-Length: 30

<h1>Hello from port 8000</h1>

You just spoke HTTP yourself. Browsers, curl, and the requests library from Python Lesson 6 all do exactly this underneath, with more headers. Leave the server running for the first case below.

Break it on purpose

Three cases. Work out the cause before you read the explanation.

Two programs, one door. With your server still running, start a second one on the same port, in a third terminal:

python3 -m http.server 8000

OSError: [Errno 48] Address already in use

That's the message on a Mac; on Linux the number is 98. Only one program can listen on a port at a time, so the second one isn't allowed to start. This is one of the most common errors in development: an old copy of your app is still running, or another app already uses the port. lsof -nP -iTCP:8000 -sTCP:LISTEN shows you who has it. Stop that program, or start yours on another port, like python3 -m http.server 8001.

Three ways to fail. Try these one after another, with the server running:

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/nope
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8001/

-w '%{http_code}\n' tells curl to print only the status code. The first prints 404; the second prints 000, because no HTTP answer came back at all. These are three different failures, and each points somewhere different:

• Timeout, like Lesson 1's port 12345: nothing answered. Check the address, the network, and firewalls in Lesson 8.
• Connection refused, like port 8001: the machine answered, but no program is listening on that port. Check the program is running, and on the port you think.
• 404: the program answered, but has nothing at that path. Check the path.

Each one gets you a step further than the one before. Knowing which you have is half of fixing it, and Lesson 12 turns that into a checklist.

It works for me, but not for you. You start a server with python3 -m http.server 8000 --bind 127.0.0.1, and it works on your laptop. A friend on the same Wi-Fi tries http://192.168.1.23:8000/ and gets refused. Why? --bind 127.0.0.1 tells the server to listen only on localhost, the address that means "this computer" from Lesson 2, so connections from anywhere else aren't heard. That's a sensible safe default for something you're testing, and a classic surprise when a program in a container or on a server listens on 127.0.0.1 and nobody can reach it. To accept connections from other machines, a program listens on 0.0.0.0, meaning every address the machine has: the * you saw in lsof.

When you're done, stop the server with Ctrl+C.

Where you are now

You know how data gets to the right machine (an address), to the right program (a port), reliably (TCP) or quickly (UDP), and what the conversation inside looks like (HTTP). There's one step from Lesson 1 left to open up: the very first one, where a name like api.civicgrid.org became an address. Lesson 5 is DNS, and it uses UDP on port 53.

Check yourself

Key terms

• Port: a number from 0 to 65,535 that picks out one program on a machine.
• Listen: what a program does when it waits for connections on a port.
• TCP (Transmission Control Protocol): reliable, ordered delivery, set up with a handshake. Used by websites, APIs, SSH and databases.
• UDP (User Datagram Protocol): fast, no handshake, no resending. Used by DNS, video calls and games.
• Handshake: the three-step exchange that opens a TCP connection.
• Protocol: a set of rules for one kind of conversation, like HTTP for the web.
• HTTP request: a method, like GET, a path, like /, and headers.
• Status code: the three-digit result in an HTTP response, like 200, 404 or 500.

Go deeper

• MDN's guide to an overview of HTTP (opens in a new tab), and its full list of status codes (opens in a new tab).
• Python's documentation for http.server (opens in a new tab), the server you ran in this lesson.