TLS and certificates

What TLS does

TLS (Transport Layer Security) is what puts the S in HTTPS. It gives a connection three things:

• Confidentiality: what's sent is encrypted, so anyone in between sees noise.
• Integrity: anything changed in transit is detected.
• Authentication: the client knows it's talking to the real server, not something pretending to be it.

The first two come from keys the client and server agree at the start of each connection, the handshake. The third comes from the server's certificate: a file saying "this public key belongs to these names", signed by a certificate authority (CA) the client already trusts. Without that third part, the first two protect you perfectly from everyone except whoever you're actually talking to.

1. A real website's certificate

openssl s_client connects and prints what the server sends. From a small probe pod in the cluster:

openssl s_client -connect kwamib.dev:443 -servername kwamib.dev </dev/null

• -servername sends the name you want (SNI, Server Name Indication), so a server hosting many sites knows which certificate to send.
• </dev/null closes the connection straight after the handshake.

 0 s:CN=kwamib.dev
   i:C=US, O=Let's Encrypt, CN=YR2
 1 s:C=US, O=Let's Encrypt, CN=YR2
   i:C=US, O=ISRG, CN=Root YR
 2 s:C=US, O=ISRG, CN=Root YR
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Verify return code: 0 (ok)

A chain. Each certificate's subject (s:) was signed by the next one's issuer (i:): the site by Let's Encrypt's YR2, YR2 by ISRG's Root YR, and that by ISRG Root X1, a root built into every operating system and browser. The client follows the chain until it reaches a certificate it already trusts. 0 (ok) means it did, over TLS 1.3.

The site's own certificate, in detail:

subject=CN=kwamib.dev
issuer=C=US, O=Let's Encrypt, CN=YR2
notBefore=Sep 17 21:09:02 2026 GMT
notAfter=Dec 16 21:09:01 2026 GMT
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Server Authentication
X509v3 Subject Alternative Name:
    DNS:kwamib.dev

• notBefore and notAfter: valid for 90 days. Let's Encrypt certificates are short on purpose, and renewed automatically.
• Subject Alternative Name (SAN): the names this certificate is good for. Browsers check this, not the subject's CN.
• Extended Key Usage: what it may be used for: identifying a web server, nothing else.

What it proves: whoever answered holds the private key for kwamib.dev, and Let's Encrypt checked that they control the domain. What it doesn't prove: who they are, or that the site is safe. A phishing site gets a valid certificate for its own name just as easily.

2. The cluster's own TLS

Everything in Kubernetes talks to the API server over TLS. Its certificate:

subject=CN=kube-apiserver
issuer=CN=kubernetes
notBefore=Nov  4 23:09:31 2026 GMT
notAfter=Nov  4 23:14:31 2027 GMT
X509v3 Subject Alternative Name:
    DNS:gitops-control-plane, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, IP Address:10.96.0.1, IP Address:172.18.0.2, IP Address:127.0.0.1

Signed by a CA called kubernetes, which kubeadm made when the cluster was built, for one year, with every name and address a client might use. And checked like a website's:

Verify return code: 21 (unable to verify the first certificate)

Not trusted. The cluster's CA isn't in any operating system's list, so a client can't follow the chain to anything it knows. Every pod is given the CA's certificate for exactly this reason, next to its service account token, and every namespace has a copy in a ConfigMap, kube-root-ca.crt:

$ ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt
namespace
token

$ openssl s_client -connect kubernetes.default.svc:443 -CAfile /tmp/k8s-ca.crt </dev/null
Verify return code: 0 (ok)

That's a private CA: one that only the clients you give it to will trust. It's the normal way to do TLS inside a system, and it's what the bookshop gets next.

3. The bookshop today

http://  200
https:// 000

Plain HTTP only. Anything between the client and the bookshop (another pod, a node, anything on the path) can read and change every request.

4. cert-manager and a practice CA

cert-manager is the standard Kubernetes tool for certificates. You describe the certificate you want as a Certificate; cert-manager gets it from an issuer, stores it in a Secret, and replaces it before it expires. Issuers can be Let's Encrypt, a company CA, Vault, a cloud CA, or a CA it keeps itself.

The change went in as one pull request, the same way as everything else in this repository:

A  apps/tls/practice-ca.yaml
M  argocd/bookshop-staging.yaml
A  argocd/cert-manager.yaml
A  argocd/tls.yaml
A  charts/bookshop/templates/certificate.yaml
M  charts/bookshop/templates/deployment.yaml
M  charts/bookshop/templates/service.yaml
A  charts/bookshop/templates/tls-conf.yaml
M  charts/bookshop/values-staging.yaml
M  charts/bookshop/values.yaml

cert-manager itself, as an Argo CD Application from its Helm chart, version v1.20.2, with crds.enabled: true so its own resource types come with it.

A practice CA, in apps/tls/practice-ca.yaml, built in three steps:

# 1. An issuer that signs a certificate with its own key. Used once, for the CA.
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
---
# 2. The root CA's certificate and private key, in the Secret practice-root-ca.
kind: Certificate
metadata:
  name: practice-root-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: gitops-practice root CA
  secretName: practice-root-ca
  duration: 87600h   # 10 years
  ...
---
# 3. practice-ca: signs certificates for apps, with the root's key.
kind: ClusterIssuer
metadata:
  name: practice-ca
spec:
  ca:
    secretName: practice-root-ca

• isCA: true makes this certificate one that can sign others.
• Ten years, because every client has to be given the root and trust it; it should change rarely.
• A ClusterIssuer can be used from any namespace (an Issuer only from its own).

The bookshop's certificate, as a new chart template, switched on only by tls.enabled:

kind: Certificate
metadata:
  name: {{ .Release.Name }}
spec:
  secretName: {{ .Release.Name }}-tls
  duration: 24h
  renewBefore: 8h
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always
  dnsNames:
    - {{ .Release.Name }}
    - {{ .Release.Name }}.{{ .Release.Namespace }}
    - {{ .Release.Name }}.{{ .Release.Namespace }}.svc
    - {{ .Release.Name }}.{{ .Release.Namespace }}.svc.cluster.local
  issuerRef:
    name: practice-ca
    kind: ClusterIssuer

• dnsNames: every name a client in the cluster might use for the Service.
• 24 hours, renewed 8 hours before the end: short on purpose, so renewal happens often enough to see. Real internal certificates are often days or weeks.
• rotationPolicy: Always: a new private key with every certificate, not the same key forever.

nginx's HTTPS server, a second config file next to the default one, with the certificate mounted from the Secret:

server {
  listen 443 ssl;
  ssl_certificate     /etc/nginx/tls/tls.crt;
  ssl_certificate_key /etc/nginx/tls/tls.key;
  ssl_protocols       TLSv1.2 TLSv1.3;
  root  /usr/share/nginx/html;
  index index.html;
}

And three settings in values-staging.yaml and the Application, so staging waits for cert-manager instead of failing: tls.enabled: true, SkipDryRunOnMissingResource (the Certificate type doesn't exist until cert-manager is installed) and a retry with backoff.

Before committing, the script rendered the chart with both environments' values. Staging's had a Certificate, port 443 and the Secret mounted. Production's check (grep -c 'Certificate|443') said 1, which was alarming for a moment, until printing the line showed it was a checksum that happens to contain 443: checksum/page: 8fc8...da443dd.... A count tells you something matched, not what.

5. Argo CD installs it

After the merge:

  cert-manager      Synced Healthy       after 15s
  tls               Synced Healthy       after 35s
  bookshop-staging  Synced Healthy       after 41s

NAME          READY   AGE
practice-ca   True    10s
selfsigned    True    10s

NAMESPACE          NAME               READY   SECRET             AGE
bookshop-staging   bookshop           True    bookshop-tls       6s
cert-manager       practice-root-ca   True    practice-root-ca   10s

HTTPS in staging 41 seconds after the merge, in the right order, because of the retries. What cert-manager made:

kubernetes.io/tls
ca.crt
tls.crt
tls.key

subject=
issuer=CN=gitops-practice root CA
notBefore=Nov  7 21:41:09 2026 GMT
notAfter=Nov  8 21:41:09 2026 GMT
serial=2BA949493ECBDF2F9D3F04AFC972BFABD6B756B6
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Basic Constraints: critical
    CA:FALSE
X509v3 Subject Alternative Name: critical
    DNS:bookshop, DNS:bookshop.bookshop-staging, DNS:bookshop.bookshop-staging.svc, DNS:bookshop.bookshop-staging.svc.cluster.local

notAfter:    2026-11-08T21:41:09Z
renewalTime: 2026-11-08T13:41:09Z

• The Secret holds three things: the certificate, its private key, and ca.crt, the CA's certificate, for clients to trust.
• subject= is empty, and the names are all in the SAN, marked critical. That's how certificates are meant to be now: the subject's CN is a leftover nothing should rely on.
• CA:FALSE: this certificate can't sign others, so stealing its key doesn't let anyone make more.
• renewalTime: cert-manager will replace it 8 hours before it expires, by itself.

The root, for comparison: CA:TRUE, issued by itself, valid until 2036. Its private key is in the Secret practice-root-ca, and anyone who reads that Secret can sign a certificate for any name every client trusts. It's now the most sensitive Secret in the cluster; Lesson 5 encrypted it at rest, and Lesson 9 limits who can read it.

6. The bookshop, as a client sees it

$ curl https://bookshop.bookshop-staging.svc/
curl: (60) SSL certificate problem: unable to get local issuer certificate

$ curl --cacert /tmp/practice-ca.crt https://bookshop.bookshop-staging.svc/
200 0

"Unable to get local issuer certificate" is the most common TLS error there is, and it means exactly this: the chain leads to a CA the client doesn't have. The fix is to give the client the CA, never to switch checking off (curl -k, verify=False, InsecureSkipVerify), which throws away the authentication and leaves only encryption to whoever answers.

 0 s:
   i:CN=gitops-practice root CA
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Verify return code: 0 (ok)

tls1_1  no protocols available
tls1_2  New, TLSv1.2, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
tls1_3  New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384

TLS 1.2 and 1.3 both work. The TLS 1.1 line needs care: no protocols available comes from the client. The probe's OpenSSL 3.5 won't offer TLS 1.1 at all any more, so this test never reached the server. nginx refuses 1.1 here because of ssl_protocols TLSv1.2 TLSv1.3, but this test doesn't prove it. (Testing old protocols takes an old client, or a tool like testssl.sh that brings its own.)

Plain HTTP on port 80 still answers. On a real service, you'd redirect it to HTTPS, or turn it off.

7. Renewal: the first test got it wrong

The plan: replace the certificate, then check that nginx serves the new one. The first version of the script deleted the Secret (cert-manager issues a new certificate at once), restarted the Deployment, and checked:

serial in the new Secret:     serial=226E4C22782E15D201804E040ACFBFC216DBDD2D
serial in the pod's file:
serial nginx is serving:      serial=2BA949493ECBDF2F9D3F04AFC972BFABD6B756B6

$ kubectl rollout restart deployment/bookshop -n bookshop-staging && kubectl rollout status deployment/bookshop -n bookshop-staging --timeout=120s
deployment "bookshop" successfully rolled out
serial nginx is serving:      serial=2BA949493ECBDF2F9D3F04AFC972BFABD6B756B6

Even after a restart, the old certificate. That would have made a dramatic lesson, and it was wrong. Two mistakes:

• It looked at the wrong pod. The merge had started a rollout a moment earlier, so two bookshop pods existed: the old one, without HTTPS, shutting down, and the new one. The script picked "the first pod labelled app=bookshop", which was the old one: no certificate file to read (the empty line), and earlier, nothing listening on 443.
• It checked too soon. rollout status returns once the new pod is ready. The old pod is still shutting down: the bookshop's preStop keeps it answering for 5 seconds, and the Service can still send it traffic for a moment.

A third look, at every pod behind the Service, showed everything serving the new certificate three minutes later. So the test ran again, properly: one steady pod, chosen by what the Service actually sends traffic to, timed from the moment the certificate changed.

8. Renewal, measured

pod behind the Service:  bookshop-58789b6f77-qkrm5
Secret:                  226E4C22782E15D201804E040ACFBFC216DBDD2D
file in the pod:         226E4C22782E15D201804E040ACFBFC216DBDD2D
served:                  226E4C22782E15D201804E040ACFBFC216DBDD2D

$ kubectl delete secret bookshop-tls -n bookshop-staging
+1s  new Secret:        685C9918BAD8A7197DCD1E4B403FE6BA9D97C621
+30s  file in the pod:   685C9918BAD8A7197DCD1E4B403FE6BA9D97C621
+30s  served:            226E4C22782E15D201804E040ACFBFC216DBDD2D
(waiting another 60 seconds, in case nginx notices by itself)
+90s  served:            226E4C22782E15D201804E040ACFBFC216DBDD2D

• +1 second: cert-manager noticed the Secret was gone and issued a new certificate.
• +30 seconds: the kubelet updated the mounted file in the pod. Secrets mounted as volumes are refreshed periodically (usually within a minute or two); environment variables and subPath mounts never are.
• +90 seconds: nginx was still serving the old certificate, a minute after the new one was on disk. nginx reads its certificate when it starts or reloads, and keeps it in memory.

$ kubectl rollout restart deployment/bookshop -n bookshop-staging && kubectl rollout status deployment/bookshop -n bookshop-staging --timeout=120s
deployment "bookshop" successfully rolled out
+91s  rollout status returned; served right away: 226E4C22782E15D201804E040ACFBFC216DBDD2D

$ kubectl wait --for=delete pod/bookshop-58789b6f77-qkrm5 -n bookshop-staging --timeout=120s
pod/bookshop-58789b6f77-qkrm5 condition met
+97s  old pod deleted; behind the Service: bookshop-6c9f59c554-gx6qk
+98s  served:            685C9918BAD8A7197DCD1E4B403FE6BA9D97C621

The same trap, caught this time: right after rollout status, still the old certificate, from the old pod finishing its shutdown. Six seconds later, with the old pod gone, the new certificate.

cert-manager renews the certificate; something else has to make the server use it. nginx needs nginx -s reload or a restart; some programs (Go's servers, Envoy, most ingress controllers) watch the file and reload by themselves. Know which kind yours is.

Spot the problem

Work out what's wrong before you read the explanation.

Renewed, and expired anyway. A team moves its internal services to cert-manager with 90-day certificates, renewed 30 days before they expire. Everything works. Ninety-one days later, half the services start failing with certificate has expired. cert-manager's status says every certificate is valid, renewed a month ago, with two months left. How? The certificates were renewed; the servers never loaded them. cert-manager put each new certificate in its Secret on day 60, and the kubelet updated the files, exactly as in this lesson. But the services read their certificate once, at startup. The ones that happened to be redeployed in the last month picked up the new one; the half that hadn't been touched in 91 days were still serving the original, from memory, after it expired. In this lesson, the same thing happened in 90 seconds instead of 91 days: new certificate on disk at +30 seconds, old one still served at +90. Three fixes: make every server reload when its certificate changes (programs that watch the file, nginx -s reload from a sidecar, or a controller such as Stakater Reloader that restarts pods when their Secret changes); monitor the certificate that's actually served, from outside, and alert well before it expires; and test renewal on purpose, by forcing one in staging, as this lesson did.

Where you are now

You can read a certificate and its chain, explain what it proves and what it doesn't, run a private CA with cert-manager, give a service HTTPS through Git, and check from a client's side that it works, including after renewal. And you've seen how easy it is to test the wrong pod at the wrong moment. The next lesson goes back to Lesson 1's findings and takes access away: least privilege in Kubernetes, starting with the two services that can read every Secret, now including the root CA's key.

Check yourself

Key terms

• TLS (Transport Layer Security): encryption, integrity and server authentication for a connection. HTTPS is HTTP over TLS.
• Certificate: a public key and the names it belongs to, signed by a CA.
• Certificate authority (CA): whoever signs certificates. Public CAs are trusted by every device; a private CA only by the clients you give it to.
• Chain: the certificates from a server's own up to a root the client trusts.
• SAN (Subject Alternative Name): the names a certificate is valid for.
• SNI (Server Name Indication): the name a client asks for at the start of the handshake.
• cert-manager: the Kubernetes controller that issues, stores and renews certificates.
• Issuer and ClusterIssuer: where cert-manager gets certificates from, for one namespace or all of them.
• ACME: the protocol Let's Encrypt and others use to check domain control and issue certificates automatically.
• mTLS (mutual TLS): both sides of a connection present certificates.

Go deeper

• cert-manager documentation (opens in a new tab), including the CA issuer and trust-manager (opens in a new tab).
• How Let's Encrypt works (opens in a new tab), and its chain of trust (opens in a new tab).
• Mozilla's SSL configuration generator (opens in a new tab), for TLS settings by server and compatibility level.
• testssl.sh (opens in a new tab), which tests a server's TLS versions and ciphers with its own client.
• Try: connect to the bookshop by the pod's IP address with --cacert. What does curl say, and which part of the certificate decides it?
• Try: add a blackbox-exporter probe for https://bookshop.bookshop-staging.svc and graph probe_ssl_earliest_cert_expiry. Then force a renewal and watch what the graph does, and doesn't, show until nginx restarts.