By the end you'll know
- A method that works on problems you've never seen before
- Which checks to run first, and how to read what they say
- How to tell the line that explains a problem from one that just agrees with it
- How to fix one thing at a time, and prove the fix from where the problem was seen
- Six real failures, and the one command that gave each away
Before you start
- You need: the bookshop server as Lessons 7 to 13 left it: nginx,
books-api, PostgreSQL, the data disk, the firewall, Sam's SSH key, and the nightly backup timer. - Where commands run: on the server, as Sam, except where it says your own computer. Start with
ssh bookshop. - Quick check, on the server:
systemctl is-active nginx books-api postgresql books-backup.timershould printactivefour times. If something isn't, the lesson that set it up is the place to start: 8 for nginx and PostgreSQL, 9 forbooks-api, 13 for the timer. - Every breakage here is undone at the end of its section. Run each section's fix before the next one.
The problem: "it's broken"
"The site's down." "It's slow." "The backup didn't run." That's how problems arrive: a symptom, often second-hand, usually urgent. The tempting move is to start changing things: restart it, reboot it, edit the setting you suspect. Sometimes that works, and you never learn why. Often it makes things worse, and now there are two problems.
Think like a doctor. A good doctor doesn't operate first. They ask where it hurts and since when, take your vital signs, run the test that tells them most, treat one thing, and check it worked. Troubleshooting a server is the same, and the vital signs are commands you already know.
It's also worth knowing where problems come from. Google's Site Reliability Engineering book puts it at roughly 70% of outages caused by a change to a live system. So the first question is almost always the same: what changed?
The method
- Say exactly what's wrong. Not "the site's broken", but "
/api/returns 502 since about 17:24, the home page is fine". What works, what doesn't, since when, and what changed around then. - See it yourself, from where it was seen. A problem only users outside see is a different problem from one you can see on the server.
- Work from the outside in. Is the server reachable? Is the service running? Is it listening? Do its logs say anything? Is the disk, memory or CPU in trouble? Each check rules out a whole layer.
- Read the evidence, with the times. Logs keep old lines. A line from yesterday doesn't explain today.
- Find the line that explains the symptom. Not one that's just consistent with it: "nginx is running" is consistent with a 502, but doesn't explain it.
- Change one thing. Then test. If you change three things and it works, you don't know which one fixed it, or what the other two broke.
- Check from where the problem was seen, and write down what you found.
What healthy looks like
You can't spot "wrong" without knowing "right". Before breaking anything, here's the bookshop working:
systemctl is-active nginx books-api postgresql books-backup.timer
curl -s -o /dev/null -w "site: %{http_code}\n" http://127.0.0.1/
curl -s -w "\napi: %{http_code}\n" http://127.0.0.1/api/ | tail -3
df -h / /srv/books-data
uptime
active
active
active
active
site: 200
{"books": ["Dune", "Beloved", "Kindred"], "served_by": "bookshop"}
api: 200
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 30G 6.2G 23G 22% /
/dev/loop0 90M 32K 83M 1% /srv/books-data
17:23:01 up 5:21, 2 users, load average: 0.00, 0.00, 0.00
And who's listening, from Lesson 9: the app on 127.0.0.1:8080, nginx on port 80, SSH on 22, PostgreSQL on 127.0.0.1:5432:
ss -tln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 5 127.0.0.1:8080 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:22 0.0.0.0:*
LISTEN 0 511 0.0.0.0:80 0.0.0.0:*
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 200 127.0.0.1:5432 0.0.0.0:*
LISTEN 0 4096 127.0.0.54:53 0.0.0.0:*
LISTEN 0 4096 [::]:22 [::]:*
Keep this in mind for each case below. On a real server, your monitoring keeps the healthy picture for you, as graphs.
Each case is a real breakage: first the command that breaks it, then the investigation. For practice, run the break line, then stop and investigate yourself before reading on. If sudo hasn't asked for Sam's password in a while, run sudo -v on its own first.
Case 1: the API says 502
sudo sed -i 's/^port = 8080/port = 8081/' /etc/books-api/override.conf && sudo systemctl restart books-api && sleep 1
The symptom:
curl -s -w "\napi: %{http_code}\n" http://127.0.0.1/api/ | tail -4
systemctl is-active books-api
</body>
</html>
api: 502
active
502 Bad Gateway means nginx is fine, but couldn't get an answer from the program behind it (Lesson 9). And systemd says that program is active. Running is not the same as working. Ask nginx what happened, and ask the system who's listening:
sudo tail -1 /var/log/nginx/error.log
ss -tln | grep -E ':808[01]'
sudo journalctl -u books-api -n 1 --no-pager -o cat
2026/10/30 17:24:50 [error] 3398#3398: *29 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: books.example.com, request: "GET /api/ HTTP/1.1", upstream: "http://127.0.0.1:8080/api/", host: "127.0.0.1"
LISTEN 0 5 127.0.0.1:8081 0.0.0.0:*
INFO books-api listening on port 8081
Two facts that disagree: nginx sends to 8080, the app listens on 8081. The time on nginx's line, 17:24:50, matches the request just made: this log also has older Connection refused lines from Lessons 8 and 9, which explain nothing today. Where does 8081 come from?
sudo grep -n port /etc/books-api/app.conf /etc/books-api/override.conf
/etc/books-api/app.conf:3:port = 8080
/etc/books-api/app.conf:6:database_port = 5432
/etc/books-api/override.conf:3:port = 8081
Lesson 3's trap, back again: the port is set twice, and override.conf wins. Fix and check:
sudo sed -i 's/^port = 8081/port = 8080/' /etc/books-api/override.conf && sudo systemctl restart books-api && sleep 1
curl -s -o /dev/null -w "after the fix: %{http_code}\n" http://127.0.0.1/api/
after the fix: 200
Case 2: the app won't start
sudo chmod 600 /etc/books-api/override.conf && sudo systemctl restart books-api; sleep 8
The symptom:
systemctl status books-api --no-pager -n 0
● books-api.service - Bookshop API (practice)
Loaded: loaded (/etc/systemd/system/books-api.service; enabled; preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Fri 2026-10-30 17:24:57 UTC; 1s ago
Process: 13208 ExecStart=/usr/local/bin/books-api (code=exited, status=1/FAILURE)
Main PID: 13208 (code=exited, status=1/FAILURE)
CPU: 74ms
activating (auto-restart): it crashed, and systemd is about to try again. Restart=on-failure (Lesson 7) restarts it every 3 seconds, forever. Useful when a crash is a one-off; here, it hides a problem behind a loop. The reason is in the journal:
sudo journalctl -u books-api --no-pager -n 12 -o cat
PermissionError: [Errno 13] Permission denied: '/etc/books-api/override.conf'
books-api.service: Main process exited, code=exited, status=1/FAILURE
books-api.service: Failed with result 'exit-code'.
books-api.service: Scheduled restart job, restart counter is at 2.
Started books-api.service - Bookshop API (practice).
Traceback (most recent call last):
File "/usr/local/bin/books-api", line 8, in <module>
for line in open(path):
^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/etc/books-api/override.conf'
books-api.service: Main process exited, code=exited, status=1/FAILURE
books-api.service: Failed with result 'exit-code'.
A Python traceback: the program's own account of where it stopped. Read it from the bottom: the last line is the error, the lines above say where. It names the exact file. Check the file, and prove the theory by reading it as the app's own user:
ls -l /etc/books-api/
sudo -u bookshop cat /etc/books-api/override.conf
total 8
-rw-r----- 1 root bookshop 151 Oct 30 01:34 app.conf
-rw------- 1 root bookshop 75 Oct 30 17:24 override.conf
cat: /etc/books-api/override.conf: Permission denied
override.conf lost its group read (Lesson 5): -rw-------, where app.conf is still -rw-r-----. The time, 17:24, says it was changed just now. sudo -u bookshop tests as the user the service really runs as, which is the only test that counts: as root, you'd have read the file fine. Fix and check:
sudo chmod 640 /etc/books-api/override.conf && sudo systemctl restart books-api && sleep 1
systemctl is-active books-api; curl -s -o /dev/null -w "after the fix: %{http_code}\n" http://127.0.0.1/api/
active
after the fix: 200
Case 3: last night's backup failed
sudo dd if=/dev/zero of=/srv/books-data/backups/.cache.tmp bs=1M status=none 2>/dev/null; true
sudo systemctl start books-backup.service; echo "start gave $?"
dd writes zeros until the disk is full: something has filled the data disk. The second line runs the backup now, instead of waiting for 2:30am.
Job for books-backup.service failed because the control process exited with error code.
See "systemctl status books-backup.service" and "journalctl -xeu books-backup.service" for details.
start gave 1
This is why Lesson 13 used a systemd timer: a failure you can see. The journal says what:
sudo journalctl -u books-backup.service --no-pager -n 6 -o cat
df -h /srv/books-data
pam_unix(runuser:session): session opened for user postgres(uid=108) by (uid=0)
pam_unix(runuser:session): session closed for user postgres
gzip: stdout: No space left on device
books-backup.service: Main process exited, code=exited, status=1/FAILURE
books-backup.service: Failed with result 'exit-code'.
Failed to start books-backup.service - Back up the books database.
Filesystem Size Used Avail Use% Mounted on
/dev/loop0 90M 88M 0 100% /srv/books-data
No space left on device, and df agrees: 100%. A full disk is the symptom; the cause is whatever filled it. Find it, from Lesson 10:
sudo du -ah /srv/books-data | sort -h | tail -4
sudo ls -la /srv/books-data/backups
16K /srv/books-data/lost+found
88M /srv/books-data
88M /srv/books-data/backups
88M /srv/books-data/backups/.cache.tmp
total 89772
drwx------ 2 root root 4096 Oct 30 17:25 .
drwxr-xr-x 4 bookshop bookshop 4096 Oct 30 16:36 ..
-rw------- 1 root root 873 Oct 30 16:48 books-20261030-164809.sql.gz
-rw-r--r-- 1 root root 91914240 Oct 30 17:25 .cache.tmp
An 88M file called .cache.tmp. It starts with a dot, so ls without -a wouldn't show it (Lesson 2); du counts everything. Two more things worth seeing: the only backup is the last good one, from 16:48, untouched, and there's no .partial file. The script stopped at the failure, and its trap cleaned up, exactly as Lesson 13 promised. Fix and check:
sudo rm /srv/books-data/backups/.cache.tmp && sudo systemctl start books-backup.service && sudo journalctl -u books-backup.service --no-pager -n 1 -o cat
df -h /srv/books-data
Finished books-backup.service - Back up the books database.
Filesystem Size Used Avail Use% Mounted on
/dev/loop0 90M 36K 83M 1% /srv/books-data
On a real server, find out what wrote that file before deleting it, or it will be back tonight: lsof on the file if it's still open (Lesson 10), and the logs around its time.
Case 4: the server is slow
sudo systemd-run --unit=report-job --quiet sh -c 'yes > /dev/null & yes > /dev/null & wait'
sleep 30
systemd-run starts a command as a temporary service. This one runs two copies of yes, which prints y as fast as it can, forever. Stand-ins for a report job gone wrong. Take the vital signs:
uptime
nproc
top -b -n 1 | head -12
18:13:01 up 6:06, 2 users, load average: 0.80, 0.20, 0.06
2
top - 18:13:02 up 6:06, 2 users, load average: 0.80, 0.20, 0.06
Tasks: 217 total, 3 running, 214 sleeping, 0 stopped, 0 zombie
%Cpu(s): 27.3 us, 72.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 1956.6 total, 391.8 free, 345.2 used, 1350.8 buff/cache
MiB Swap: 3903.0 total, 3903.0 free, 0.0 used. 1611.4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13465 root 20 0 5256 1664 1664 R 100.0 0.1 0:30.18 yes
13466 root 20 0 5256 1664 1664 R 90.0 0.1 0:30.17 yes
1 root 20 0 22484 12824 8600 S 0.0 0.6 0:03.79 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
- The load average is three numbers: the average number of processes running or waiting for a CPU over the last 1, 5 and 15 minutes. Compare it with
nproc, the number of CPUs: here, 2. Load near the CPU count means busy; well above means queueing. And notice it's only 0.80 after 30 seconds of both CPUs flat out: the 1-minute average is still catching up. Load lags.topdoesn't. 0.0 id: no idle CPU at all.usis time running programs' own code,sytime in the kernel, here mostly writing to/dev/null.- Two processes at about 100% each, both
yes, owned by root, andTIME+says they've used 30 seconds of CPU each.
Memory is fine: avail Mem is 1.6 GB and no swap is used (Lesson 10). So it's CPU, and it's these two. Who started them? Killing a process without knowing is how you stop a job someone needed, or get it straight back if something restarts it:
ps -eo pid,user,%cpu,etime,cmd --sort=-%cpu | head -4
systemctl status $(pgrep -o yes) --no-pager -n 0 | head -3
PID USER %CPU ELAPSED CMD
13466 root 99.8 00:30 yes
13465 root 99.8 00:30 yes
13350 root 0.3 28:06 [kworker/0:2-events]
● report-job.service - /usr/bin/sh -c "yes > /dev/null & yes > /dev/null & wait"
Loaded: loaded (/run/systemd/transient/report-job.service; transient)
Transient: yes
systemctl status with a process ID names the service it belongs to: report-job. ELAPSED says when it started, which is the "what changed?" question answered. Stop the service, not the processes, and both go:
sudo systemctl stop report-job
sleep 2; ps -eo pid,%cpu,cmd --sort=-%cpu | head -3
PID %CPU CMD
13350 0.3 [kworker/0:2-events]
13167 0.2 [kworker/1:0-events]
Case 5: an nginx change takes the site down
A one-character change: the ; at the end of a line in the site's settings, gone. Keep a copy first, as you would before any edit:
sudo cp /etc/nginx/sites-available/books /tmp/books.nginx.bak
sudo sed -i 's/proxy_read_timeout 30s;/proxy_read_timeout 30s/' /etc/nginx/sites-available/books
Now apply it the gentle way, and the blunt way:
sudo systemctl reload nginx; echo "reload gave $?"
curl -s -o /dev/null -w "site after reload: %{http_code}\n" http://127.0.0.1/
sudo systemctl restart nginx; echo "restart gave $?"
curl -s -o /dev/null -w "site after restart: %{http_code}\n" http://127.0.0.1/
systemctl is-active nginx
Job for nginx.service failed.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
reload gave 1
site after reload: 200
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
restart gave 1
site after restart: 000
failed
The reload failed, and the site stayed up. nginx checked the new settings, refused them, and kept running with the old ones. The restart stopped nginx, then couldn't start it again. The site is down: 000 means no answer at all. Same mistake, very different outcome. What's wrong?
sudo journalctl -u nginx --no-pager -n 4 -o cat
sudo nginx -t
nginx: configuration file /etc/nginx/nginx.conf test failed
nginx.service: Control process exited, code=exited, status=1/FAILURE
nginx.service: Failed with result 'exit-code'.
Failed to start nginx.service - A high performance web server and a reverse proxy server.
2026/10/30 18:13:04 [emerg] 13517#13517: unexpected "}" in /etc/nginx/sites-enabled/books:8
nginx: configuration file /etc/nginx/nginx.conf test failed
nginx -t tests the settings, and names the exact file and line: unexpected "}" on line 8. The real mistake is just before it, on line 7, where the missing ; meant nginx read on into the }. Error messages often point at where the program noticed, not where the mistake is. (sites-enabled/books is the link to sites-available/books, Lesson 8.) Fix, test, start, check:
sudo cp /tmp/books.nginx.bak /etc/nginx/sites-available/books && sudo nginx -t && sudo systemctl start nginx
curl -s -o /dev/null -w "site after the fix: %{http_code}\n" http://127.0.0.1/
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
site after the fix: 200
The habit this teaches: sudo nginx -t && sudo systemctl reload nginx. Test first, reload, never restart to apply a settings change. The same pattern from Lesson 11: sshd -t, then reload.
Case 6: it works on the server, but not from your computer
sudo ufw delete allow 80/tcp
sudo ufw status
curl -s -o /dev/null -w "from the server itself: %{http_code}\n" http://127.0.0.1/
exit
Rule deleted
Rule deleted (v6)
Status: active
To Action From
-- ------ ----
OpenSSH ALLOW Anywhere
OpenSSH (v6) ALLOW Anywhere (v6)
from the server itself: 200
logout
Connection to 192.168.34.130 closed.
Fine from the server itself. The exit took you back to your own computer. See it from where users see it:
curl -s -m 5 -o /dev/null -w "from the Mac: %{http_code}\n" http://192.168.34.130/; echo "curl gave $?"
ssh -o BatchMode=yes bookshop 'echo "ssh on port 22 still works"'
from the Mac: 000
curl gave 28
ssh on port 22 still works
000 and exit code 28: curl's code for "timed out". -m 5 gave up after 5 seconds, and nothing came back at all, not even a refusal. Port 22 works, so the server is up and the network is fine. That combination, one port silent while others work, points at a firewall. A refused connection would have failed at once; a firewall that drops traffic just lets you wait. Back on the server, the kernel logs what the firewall drops:
ssh bookshop
Then on the server:
sudo journalctl -k --no-pager --since "-10min" | grep "UFW BLOCK" | grep "DPT=80" | tail -2
Oct 30 18:13:16 bookshop kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:0c:29:3d:00:9d:86:2f:57:ea:d6:65:08:00 SRC=192.168.34.1 DST=192.168.34.130 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=51545 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
Oct 30 18:13:17 bookshop kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:0c:29:3d:00:9d:86:2f:57:ea:d6:65:08:00 SRC=192.168.34.1 DST=192.168.34.130 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=51545 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
-k is the kernel's own messages. Each line is one dropped packet: SRC your computer, DST the server, DPT=80 the port it wanted, SYN a request to start a connection. Your computer asked twice, a second apart, and was ignored both times. Fix, then check from outside:
sudo ufw allow 80/tcp
exit
Rule added
Rule added (v6)
logout
Connection to 192.168.34.130 closed.
curl -s -m 5 -o /dev/null -w "from the Mac: %{http_code}\n" http://192.168.34.130/
from the Mac: 200
Checking from the server would have said 200 all along. The fix is only proved from where the problem was seen. On a cloud server, there's often a second firewall outside the server too, such as an AWS security group, which this log can't see: if traffic never reaches the server, it never appears here.
All six at a glance
| Symptom | The line that explained it | From | Cause |
|---|---|---|---|
502 on /api/ | 127.0.0.1:8081, while nginx sends to 8080 | ss -tln | A changed port |
| App won't start | PermissionError ... override.conf | journalctl -u | Lost group read |
| Backup failed | No space left on device | journalctl -u, df, du | A hidden 88M file |
| Server slow | Two yes at 100% CPU | top, systemctl status PID | A runaway job |
| Site down after a change | unexpected "}" ... books:8 | nginx -t | A missing ;, then a restart |
| Fails only from outside | [UFW BLOCK] ... DPT=80 | journalctl -k | A deleted firewall rule |
Every one was a change. Every one was found by a command from an earlier lesson. Try them in any order:
On a real server
- Restore service first, find the cause second, when users are waiting. If the last change broke it, undo the change. Then investigate, with the pressure off.
- Capture evidence before you restart. A restart clears the state you need: the process list, the open files, the error on screen. Thirty seconds of
ss,topandjournalctlfirst can save hours. - Keep a timeline as you go: times, what you checked, what you saw, what you changed. It's how others can help mid-incident, and it becomes the write-up afterwards.
- Write it up, without blame. The question is "what let this happen?", not "who did it?". Case 5's answer isn't "be more careful"; it's "always
nginx -t && reload", ideally enforced by the deploy process. - Add the check that would have caught it. Each case here suggests one: alert on 5xx responses, on a crash-looping service, on a failed backup, on a full disk, on sustained high load, and test the site from outside.
Spot the problem
Work out what's wrong before you read the explanation.
The fix that didn't stick. The data disk fills up at night. Each morning, someone deletes the big file and the disk is fine again. Every night, it fills up again. What's missing? The cause. The big file was the symptom; something writes it every night. The timestamp on the file (ls -l) says when it's written, and the logs, systemctl list-timers and /etc/cron.d around that time say what. Deleting the file was restoring service, which is right. Stopping there wasn't.
Keep this
| Symptom | Check first |
|---|---|
| Anything at all | What changed? systemctl is-active NAME, sudo journalctl -u NAME -n 50 |
| 502 or 504 from nginx | sudo tail /var/log/nginx/error.log, ss -tln, the app's port setting |
| Service won't start | systemctl status NAME, the journal from the bottom up, sudo -u USER to test as the service's user |
| Can't write, job failed | df -h, then sudo du -ah DIR | sort -h | tail |
| Slow | uptime against nproc, top, free -h, systemctl status PID |
| Settings change broke it | sudo nginx -t, sudo sshd -t, sudo visudo -c |
| Works inside, not outside | curl -m 5 from outside, sudo ufw status, sudo journalctl -k | grep "UFW BLOCK" |
| Applying a settings change | Test, then reload: sudo nginx -t && sudo systemctl reload nginx |
Where you are now
You have a method, and you've used it six times: say exactly what's wrong, see it yourself, work from the outside in, read the evidence with its times, change one thing, and prove the fix from where the problem was seen. The bookshop is back to healthy. Lesson 15 is the last: everything in the track, on a server set up from scratch and locked down, the way you'd do it for real.
Check yourself
nginx returns 502 and systemctl says the app is active. What do you check next?
A service keeps restarting every few seconds. Where is the reason most likely to be?
Why test a settings file with sudo -u bookshop cat, not just sudo cat?
The load average is 0.80 on a 2-CPU server, but top shows both CPUs at 0% idle. Which do you believe?
You broke nginx's settings. Which command keeps the site up?
curl from your computer times out, but SSH to the same server works. What's the likeliest cause?
Key terms
- Symptom and cause: what you can see going wrong, and the thing that makes it happen. Fixing a symptom brings it back.
- Baseline: what the server looks like when it's healthy, to compare against.
- 502 Bad Gateway: nginx is working, but the program behind it didn't answer.
- Traceback: a program's account of where it crashed. The error is at the bottom.
- Load average: processes running or waiting for a CPU, averaged over 1, 5 and 15 minutes.
- Timeout versus refused: no answer at all, often a firewall, against an instant "nothing's listening".
- Write-up (postmortem): the record of an incident and what will stop it happening again.
Go deeper
- Break something yourself and investigate before you look:
sudo systemctl stop postgresql, then work out what still works and what doesn't, and how you'd find out. Start it again afterwards. journalctl -p err -bshows only errors, since the last boot: a fast first look on a server you don't know.sudo ufw loggingshows how much the firewall logs. The default,low, records dropped packets like the ones in Case 6.- Google's Site Reliability Engineering book, the chapters on effective troubleshooting and on postmortem culture.
- Lesson 9, Networking from the shell, and Lesson 10, Disks, storage and memory, for the tools most of these cases used.