Resources Networking and DNS Networking basics

Networking basics · Lesson 9 of 12 · about 25 minutes

TLS and HTTPS: certificates and trust

How HTTPS keeps traffic private and proves who you're talking to: what TLS does, what happens in the handshake, what a certificate contains, how the chain of trust works, and why certificates expire. You'll inspect real certificates with curl and openssl.

By the end you'll know

  • The three jobs TLS does: privacy, tamper-proofing and proving identity
  • What happens in a TLS handshake, step by step
  • What a certificate contains, and how wildcards work
  • How the chain of trust gets from a website to something your computer already trusts
  • Why certificates expire, and what the common certificate errors mean

Where you are

Lesson 8 made sure only the right traffic gets through. But everything you've sent so far travels in the open. Your request passes through your router, your internet provider and several other networks, as Lesson 6's traceroute showed, and any of them could read it, or change it, on the way.

You've seen the fix in passing twice. Lesson 1's curl printed a certificate for *.civicgrid.org, issued by Let's Encrypt. Lesson 4 said HTTPS is HTTP sent inside an encrypted connection. This lesson opens that connection up. It's called TLS (Transport Layer Security), and it's the S in HTTPS.

Three jobs

Sending a letter in a sealed, tamper-evident envelope does two things: nobody can read it on the way, and if anybody opens it, you can tell. But it doesn't prove who's on the other end. For that, you'd check their passport, issued by a government you trust.

TLS does all three:

  • Privacy: everything is encrypted, scrambled so only the two ends can read it. Your provider can see that you connected to api.civicgrid.org, but not what you asked for or what came back.
  • Tamper-proofing: if anything in the middle changes even one bit, the receiving end notices and rejects it.
  • Proof of identity: the server shows a certificate, its passport, which proves it really is api.civicgrid.org and not someone pretending to be. Without this, encryption alone would only give you a private conversation with an impostor.

Before any of that, the two sides have to agree how to do it. That's the handshake.

Watch a handshake

Remember Lesson 4's TCP handshake, where the two sides agree to talk? TLS adds a second handshake on top, right after it. curl shows you every step with -v. This keeps only the TLS lines:

BASH
curl -sv -o /dev/null https://api.civicgrid.org/ 2>&1 | grep -E "SSL|TLS|ALPN|subject|issuer|expire|start date"

2>&1 sends curl's detail lines, which normally go to a separate stream, into the pipe too, so grep can filter them. Here's a real run:

OUTPUT
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519MLKEM768 / id-ecPublicKey
* ALPN: server accepted h2
*  subject: CN=civicgrid.org
*  start date: Sep  6 01:07:48 2026 GMT
*  expire date: Dec  5 01:07:47 2026 GMT
*  subjectAltName: host "api.civicgrid.org" matched cert's "*.civicgrid.org"
*  issuer: C=US; O=Let's Encrypt; CN=YE2
*  SSL certificate verify ok.

A few lines are trimmed. OUT is curl speaking, IN is the server. In plain words:

  1. Client hello. curl says: "I'd like to talk securely. Here are the versions and methods I support, and the name I want: api.civicgrid.org." It also offers h2 and http/1.1, the HTTP versions it can speak, which is called ALPN (Application-Layer Protocol Negotiation).
  2. Server hello. Cloudflare picks the newest version they both support, TLS 1.3, and a method, and sends its half of the key exchange, below.
  3. Certificate and CERT verify. The server sends its certificate, and proves it holds the secret key that goes with it.
  4. Finished, both ways. Each side confirms it saw the same handshake as the other, so nothing was tampered with on the way. From here, everything is encrypted.

Then curl checks the certificate, and prints its verdict: SSL certificate verify ok. The server also accepted h2, which is why Lesson 1's response came back over HTTP/2. ("SSL", Secure Sockets Layer, is TLS's old name; tools still use it, but SSL itself is long retired.)

The TLS handshakeTwo columns, curl on the left and the server on the right. First the TCP handshake from Lesson 4 opens a connection. Then curl sends a client hello with the name api.civicgrid.org and the versions it supports. The server replies with a server hello, its certificate, proof that it holds the matching key, and finished. curl checks the certificate and sends finished. After that, the HTTP request and response travel encrypted.curlServerTCP handshake, as in Lesson 4Client hello: "api.civicgrid.org, TLS 1.3, h2"Server hello, certificate, proof, finishedchecks the certificateFinishedFrom here, HTTP travels encrypted
TLS 1.3 needs one round trip on top of TCP's handshake. The name you want travels in the very first message, which is how one server can host certificates for many sites.

Agreeing a secret in public

The clever part is step 2. The two sides need to agree on a secret key to encrypt with, but every message between them can be seen by anyone in the middle. How do you agree a secret out loud?

Here's the idea in everyday terms. Imagine you post someone an open padlock, keeping the only key. They put a message in a box, snap your padlock shut, and post it back. Anyone handling the box saw the padlock, but only you can open it. That's public-key cryptography: a public key that anyone can use to lock, and a matching private key, kept secret, that's the only way to unlock.

In TLS, each side mixes its own secret with the other's public half, and both arrive at the same session key without ever sending it. Everything after the handshake is encrypted with that session key, which is thrown away when the connection ends. The X25519MLKEM768 in curl's output names the exact method. The MLKEM768 part is a new design meant to stay safe even against future quantum computers, and Cloudflare already uses it.

What's in a certificate

curl showed you parts of the certificate. openssl, which comes with macOS and Linux, shows the whole thing. This connects to the server, then prints just the parts you care about:

BASH
echo | openssl s_client -connect api.civicgrid.org:443 -servername api.civicgrid.org 2>/dev/null | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

It's a long line, but it's two commands joined by |. openssl s_client opens a TLS connection, like curl without the HTTP; echo | gives it an empty message so it closes straight away, and 2>/dev/null hides its chatter. -servername sends the name in the client hello, more on that below. The second command, openssl x509, reads the certificate that came back and prints the chosen parts.

OUTPUT
subject=CN=civicgrid.org
issuer=C=US, O=Let's Encrypt, CN=YE2
notBefore=Sep  6 01:07:48 2026 GMT
notAfter=Dec  5 01:07:47 2026 GMT
X509v3 Subject Alternative Name:
    DNS:*.civicgrid.org, DNS:civicgrid.org
  • Subject: who the certificate is for. CN means common name.
  • Issuer: who vouches for it: here Let's Encrypt, through its intermediate certificate called YE2. An organisation that issues certificates is a certificate authority, or CA.
  • notBefore and notAfter: when it's valid. This one lasts 90 days.
  • Subject Alternative Names: the full list of names it covers. This is the list that's actually checked; the subject is mostly for show.

*.civicgrid.org is a wildcard: it covers any single name in front of civicgrid.org, like api.civicgrid.org or www.civicgrid.org. That's the matched cert's "*.civicgrid.org" line from curl. A wildcard only covers one level, so v2.api.civicgrid.org wouldn't match, and it doesn't cover civicgrid.org itself, which is why that's listed separately.

Compare this site's certificate:

BASH
echo | openssl s_client -connect www.kwamib.dev:443 -servername www.kwamib.dev 2>/dev/null | openssl x509 -noout -subject -issuer -dates -ext subjectAltName
OUTPUT
subject=CN=www.kwamib.dev
issuer=C=US, O=Let's Encrypt, CN=YR1
notBefore=Sep 17 08:03:21 2026 GMT
notAfter=Dec 16 08:03:20 2026 GMT
X509v3 Subject Alternative Name:
    DNS:www.kwamib.dev

Also Let's Encrypt, also 90 days, but no wildcard: it covers exactly one name. YR1 is a different Let's Encrypt intermediate, for a different kind of key: YE ones are for the newer, smaller ECDSA (elliptic curve) keys, and YR for the older RSA kind. Both are perfectly good.

The chain of trust

Why should your computer believe Let's Encrypt? It doesn't have to know Let's Encrypt directly. Your operating system and browser come with a short list of root certificates, from a few dozen organisations that have been checked and audited. That list is called the trust store. Every certificate you accept has to lead back to one of them.

The server sends the in-between certificates, so your computer can follow the trail. Add -showcerts to see them:

BASH
echo | openssl s_client -connect api.civicgrid.org:443 -servername api.civicgrid.org -showcerts 2>/dev/null | grep -E "^ *[0-9] s:|^ *i:"
OUTPUT
 0 s:CN=civicgrid.org
   i:C=US, O=Let's Encrypt, CN=YE2
 1 s:C=US, O=Let's Encrypt, CN=YE2
   i:C=US, O=ISRG, CN=Root YE
 2 s:C=US, O=ISRG, CN=Root YE
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X2
 3 s:C=US, O=Internet Security Research Group, CN=ISRG Root X2
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1

Each pair is one certificate: s: is its subject and i: is who issued it. Read it top down, and each issuer is the next certificate's subject:

  • 0 is CivicGrid's certificate, signed by YE2.
  • 1 is YE2, an intermediate, signed by Root YE.
  • 2 and 3 link Root YE back to older Let's Encrypt roots, ISRG Root X2 and X1. ISRG (Internet Security Research Group) is the non-profit behind Let's Encrypt.

Your computer follows the chain until it reaches a root that's already in its trust store, and stops there. Root YE is part of a new set Let's Encrypt is rolling out; the extra links to its older roots are there so that computers which don't know the new root yet still find one they do. Each certificate carries a signature from its issuer, made with the issuer's private key and checked with its public key, so no link can be forged.

In my real setup

The certificate you just inspected is the one Cloudflare shows visitors, renewed automatically every few months, and I never touch it. But Cloudflare sits in the middle: it ends your TLS connection, reads the request, and opens a second, separate connection to my servers. That second connection needs securing too, and it's a separate setting. TLS ending at a middle server like this is called TLS termination, and Lesson 10 covers it.

Why certificates expire

Both certificates you inspected last 90 days. That's on purpose. If a certificate's private key is ever stolen, the thief can pretend to be the site until the certificate expires, so shorter lives limit the damage. The industry is shortening them further: the rules browsers and CAs agree on will bring the maximum down in stages, to 47 days by 2029.

Nobody renews certificates by hand at that pace. Let's Encrypt, and most hosts and CDNs, use a protocol called ACME (Automatic Certificate Management Environment): a program on your server proves to the CA that it controls the domain, gets a new certificate, and installs it, every couple of months, with no one involved. Vercel does it for this site; Cloudflare does it for CivicGrid's. In Kubernetes, a tool called cert-manager does the same job.

The flip side: when automatic renewal quietly breaks, the site works perfectly until the day the certificate expires, and then it doesn't work at all. Which brings us to the errors.

Break it on purpose

Three cases. Work out the cause before you read the explanation.

An expired certificate. badssl.com is a public test site, run for exactly this purpose, with deliberately broken certificates on different names:

BASH
curl -sv -o /dev/null https://expired.badssl.com/ 2>&1 | tail -3
OUTPUT
} [2 bytes data]
* SSL certificate problem: certificate has expired
* closing connection #0

curl refuses, and so would your browser, with a full-page warning. Nothing else is wrong: the server's running, the name and the chain are fine. The certificate's notAfter date has simply passed. When a site that worked yesterday suddenly shows certificate errors for everyone, check its dates with the openssl x509 -dates command above. It's almost always a renewal that silently stopped. Checking how many days are left, and alerting before it's zero, is a good job for the Python checker from the Python track.

The right server, the wrong name. Ask Cloudflare's address for a site it doesn't host:

BASH
curl -sv -o /dev/null --resolve wrong.example.test:443:104.21.69.165 https://wrong.example.test/ 2>&1 | tail -4
OUTPUT
* TLSv1.3 (IN), TLS alert, handshake failure (552):
{ [2 bytes data]
* TLS connect error: error:0A000410:SSL routines::ssl/tls alert handshake failure
* closing connection #0

--resolve from Lesson 5 sends the request to Cloudflare's address, but the client hello asks for wrong.example.test. That name in the client hello is called SNI (Server Name Indication), and it's how one address can serve certificates for thousands of sites: the server picks the certificate by the name you ask for. Cloudflare has no certificate for that name, so it ends the handshake on the spot. In Lesson 1 you saw the other version of this: Vercel did send a certificate, but for a different name, and curl refused it as a mismatch. Either way, the cause is a name the server isn't set up for, often a DNS record pointing somewhere new before the certificate is ready.

"Just turn off verification." A script calling an API fails with a certificate error. Someone suggests the quick fix: curl -k, or verify=False in Python's requests library from Python Lesson 6. Now it works. What's wrong with that? Those options skip the identity check entirely. The connection is still encrypted, but to anyone at all, including an impostor in the middle, which defeats the point of the third job. It's fine for a moment's testing on your own machine, and never in a script or production code. The real fix depends on the error: renew the certificate, fix the name, or, for a company's own internal certificates, add its root to the trust store properly.

Where you are now

You can explain what TLS protects and how, read the handshake, inspect any site's certificate and chain with openssl, and recognise the common certificate errors. You also know TLS often ends before your servers do, at something in the middle, like Cloudflare. Lesson 10 is about those middle pieces: load balancers and reverse proxies, which share traffic across servers and are often where TLS is terminated.

Check yourself

  1. Which of these does TLS's certificate check protect you from?

  2. Does a certificate for *.civicgrid.org cover api.civicgrid.org?

  3. Why does your computer trust a certificate from Let's Encrypt?

  4. What does SNI do?

  5. A site that worked yesterday shows certificate errors for everyone. What's most likely?

  6. Why is verify=False a bad fix for a certificate error?

Key terms

  • TLS (Transport Layer Security): encryption, tamper-proofing and proof of identity for a connection. The S in HTTPS.
  • Handshake: the exchange at the start where both sides agree how to talk and the server proves who it is.
  • Public and private keys: a pair where anyone can lock with the public key, and only the private key unlocks.
  • Certificate: a server's proof of identity, listing the names it covers, its dates, and who issued it.
  • Certificate authority (CA): an organisation that issues certificates, like Let's Encrypt.
  • Chain of trust: certificates signing each other, back to a root in your trust store.
  • SNI (Server Name Indication): the name sent in the client hello, so the server picks the right certificate.
  • ACME (Automatic Certificate Management Environment): the protocol that renews certificates automatically.

Go deeper

From www.kwamib.dev/resources/networking/tls-and-https