A real server: set it up and lock it down

The problem: the server nobody can rebuild

Over eight lessons, the bookshop server was built by hand: a user here, a permission there, a setting changed, a fix made at 5pm. It works. Now imagine it's gone tonight: the disk fails, or the cloud account is lost. Could you build it again, exactly? Every setting? From memory?

That's a snowflake server: unique, and impossible to recreate. Most companies have some, and they're the servers people are afraid to touch. The fix is to stop treating the server as the thing you keep. Keep the recipe instead. A cook who's made a dish a hundred times can't tell you every pinch; a written recipe gives you the same dish every time, in anyone's kitchen. A setup script is the recipe, and the server is just today's dish. If it goes wrong, throw it away and cook it again.

This lesson writes the bookshop's recipe, builds a second server from it, and proves the result is locked down. The first server stays exactly as it is.

Capture what was built

The recipe needs the bookshop's real files, copied from the real server, not retyped from memory. On your own computer:

lp
mkdir -p fresh/files
vagrant ssh -c 'cat /usr/local/bin/books-api' > fresh/files/books-api
vagrant ssh -c 'sudo cat /etc/books-api/app.conf' > fresh/files/app.conf
vagrant ssh -c 'cat /etc/nginx/sites-available/books' > fresh/files/nginx-books

The same for override.conf, the two unit files, the backup script and its timer. Then your public key, which is what will let Sam in:

cp ~/.ssh/bookshop_ed25519.pub fresh/files/sam.pub

Check what you captured. Two of these went wrong, and nothing said so:

cat: /etc/nginx/sites-available/books: Permission denied^M$
# Settings for the practice app^M$
name = books-api^M$

That's cat -A, which shows invisible characters: $ is the end of a line, and ^M is a carriage return, \r. Every file had one on every line. vagrant ssh -c gives the command a terminal, and a terminal ends lines with \r\n, the Windows way. A script with \r in it breaks: #!/usr/bin/env python3\r names a program that doesn't exist. And the nginx file was an error message, saved as if it were the file: vagrant can't read it without sudo. Two fixes: sed -i 's/\r$//' fresh/files/* strips the carriage returns, and sudo cat reads the nginx file. After that, app.conf was 151 bytes, the same as ls -l showed on the server in Lesson 14. Always check a copy before you trust it.

The setup script

setup.sh (opens in a new tab) is about 140 lines, in steps that follow the track: packages and automatic updates, the two users, sudo, SSH, the firewall, the data disk, the database, the app, nginx, and the backup timer. It starts the way Lesson 13 says every script should, set -euo pipefail, and stops unless it's run as root.

The one rule that makes it a recipe: every step checks first, and only changes what's different. Run it on a fresh server and it builds everything. Run it again and it changes nothing. Run it on a server someone has tinkered with, and it puts back exactly what was changed. That property has a name, idempotent: doing it twice has the same effect as doing it once. The heart of it is one small function:

die() { echo "setup.sh: $*" >&2; exit 1; }

# put SRC DEST MODE OWNER:GROUP
# Installs a file only if it's missing or different.
# Returns 0 if it changed the file, 1 if it was already right. On any error it
# stops the whole script with exit, not return: put is always called inside a
# condition (if, &&, ||), where set -e is switched off, so a failure inside it
# would otherwise be ignored and reported as "installed".
put() {
	local src=$1 dest=$2 mode=$3 owner=$4
	[[ $mode =~ ^0?[0-7]{3}$ ]] || die "put: mode must look like 640 or 0640, not '$mode'"
	mode=${mode#0}  # stat -c %a prints 640, never 0640
	[[ $owner == *:* ]] || die "put: owner must be USER:GROUP, not '$owner'"
	[ -r "$src" ] || die "put: can't read $src"
	if [ -f "$dest" ] && cmp -s -- "$src" "$dest" \
		&& [ "$(stat -c %a -- "$dest")" = "$mode" ] \
		&& [ "$(stat -c %U:%G -- "$dest")" = "$owner" ]; then
		echo "    unchanged: $dest"
		return 1
	fi
	install -m "$mode" -o "${owner%:*}" -g "${owner#*:}" -- "$src" "$dest" \
		|| die "put: couldn't install $dest"
	echo "    installed: $dest"
	return 0
}

• cmp -s compares two files byte by byte, silently: its exit status says whether they're identical.
• stat -c %a is the mode as a number, 640, and %U:%G the owner and group. A file with the right contents but the wrong permissions counts as different. stat prints 640, never 0640, so put drops a leading 0; otherwise 0640 would never match, and the file would be reinstalled on every run. (stat -c is the Linux version. macOS's stat takes different options, which doesn't matter here: this script only runs on the server.)
• install (Lesson 11) copies a file with its mode and owner set in one go. ${owner%:*} is the part before the :, ${owner#*:} the part after (Lesson 12's ${f%.csv}, from the other end too).
• -- means "no more options": a path starting with - is read as a path, not an option.
• It returns 0 only when it changed something, so a step can react: restart the app only if one of its files changed. And because "unchanged" is a return of 1, put must always be called inside a condition, if put ..., put ... && changed=1 or put ... || true, or set -e would stop the script on a file that was already right.
• Errors use exit, not return. The reason is in the comment at the top, and the story is in "Three bugs" below.

Everything else follows the same pattern, in whatever form fits:

id sam > /dev/null 2>&1 || { useradd --create-home --shell /bin/bash --groups sudo sam; echo "    created: sam"; }

grep -q "db.internal" /etc/hosts || echo "127.0.0.1 db.internal" >> /etc/hosts

psql_q -d books -c "CREATE TABLE IF NOT EXISTS books (id serial PRIMARY KEY, title text NOT NULL, stock int NOT NULL DEFAULT 0);"

Make the user only if id says it doesn't exist. Add the line only if grep doesn't find it. Create the table if not exists. Insert the sample books only if the table is empty. None of it is clever: each step just asks before it acts.

Build a server from nothing

fresh/Vagrantfile is the one from Lesson 7, with a new name and one new line:

config.vm.hostname = "bookshop2"
config.vm.synced_folder ".", "/setup"
config.vm.provision "shell", path: "setup.sh"

provision tells Vagrant to run setup.sh as root on the new server the first time it starts. On a cloud server, the same job is done by cloud-init: the provider runs your script at first boot. In fresh:

cd fresh
vagrant up

The output, filtered with grep down to the script's own lines (the sudo step's heading starts with a small letter, so the filter dropped it: the 10-bookshop line under Users is that step):

    default: ==> Packages (Lesson 8)
    default:     nginx, postgresql, unattended-upgrades and shellcheck installed; system up to date
    default: ==> Automatic security updates (Lesson 8)
    default:     unchanged: /etc/apt/apt.conf.d/20auto-upgrades
    default: ==> Users (Lessons 5 and 11)
    default:     created: bookshop
    default:     created: sam
    default:     installed: /home/sam/.ssh/authorized_keys
    default:     installed: /etc/sudoers.d/10-bookshop
    default: ==> SSH: keys only, no root (Lesson 11)
    default:     installed: /etc/ssh/sshd_config.d/10-bookshop.conf
    default:     sshd settings tested and applied
    default: ==> Firewall: SSH and web only (Lesson 9)
    default: ==> Data disk at /srv/books-data (Lesson 10)
    default:     created: /var/tmp/books-data.img
    default:     mounted: /dev/loop0 80.3M
    default: ==> Database (Lessons 8 and 13)
    default:     created: database books
    default:     books table: 3 rows
    default: ==> The app (Lessons 7 and 9)
    default:     installed: /usr/local/bin/books-api
    default:     installed: /etc/books-api/app.conf
    default:     installed: /etc/books-api/override.conf
    default:     installed: /etc/systemd/system/books-api.service
    default:     books-api restarted
    default: ==> Web server (Lessons 8 and 14)
    default:     installed: /etc/nginx/sites-available/books
    default:     nginx settings tested and reloaded
    default: ==> Nightly backup (Lesson 13)
    default:     installed: /usr/local/bin/books-backup
    default:     installed: /etc/systemd/system/books-backup.service
    default:     installed: /etc/systemd/system/books-backup.timer
    default:     next backup: Sat 2026-10-31 02:30:00
    default: ==> Done
    default:     Now run: sudo /setup/audit.sh

A whole server, every lesson from 5 to 13, in a few minutes. One unchanged already: this Ubuntu image ships an automatic-updates file identical to the script's, so put left it alone. Now run it again:

vagrant provision

    default: ==> Packages (Lesson 8)
    default:     nginx, postgresql, unattended-upgrades and shellcheck installed; system up to date
    default: 
    default: ==> Automatic security updates (Lesson 8)
    default:     unchanged: /etc/apt/apt.conf.d/20auto-upgrades
    default: 
    default: ==> Users (Lessons 5 and 11)
    default:     unchanged: /home/sam/.ssh/authorized_keys
    default: 
    default: ==> sudo needs a password (Lesson 11)
    default:     unchanged: /etc/sudoers.d/10-bookshop
    default:     sudo rules check out
    default: 
    default: ==> SSH: keys only, no root (Lesson 11)
    default:     unchanged: /etc/ssh/sshd_config.d/10-bookshop.conf
    default: 
    default: ==> Firewall: SSH and web only (Lesson 9)
    default:     ufw active: OpenSSH, 80/tcp
    default: 
    default: ==> Data disk at /srv/books-data (Lesson 10)
    default:     mounted: /dev/loop0 80.3M
    default: 
    default: ==> Database (Lessons 8 and 13)
    default:     books table: 3 rows
    default: 
    default: ==> The app (Lessons 7 and 9)
    default:     unchanged: /usr/local/bin/books-api
    default:     unchanged: /etc/books-api/app.conf
    default:     unchanged: /etc/books-api/override.conf
    default:     unchanged: /etc/systemd/system/books-api.service
    default: 
    default: ==> Web server (Lessons 8 and 14)
    default:     unchanged: /etc/nginx/sites-available/books
    default: 
    default: ==> Nightly backup (Lesson 13)
    default:     unchanged: /usr/local/bin/books-backup
    default:     unchanged: /etc/systemd/system/books-backup.service
    default:     unchanged: /etc/systemd/system/books-backup.timer
    default:     next backup: Sat 2026-10-31 02:30:00
    default: 
    default: ==> Done
    default:     Now run: sudo /setup/audit.sh

Every file unchanged, no user created, nothing restarted. That's what makes it safe to run whenever you like: on a schedule, after every change to the script, or on a server you're not sure about.

Prove it: the audit

The setup script says what it did. That's not the same as proof. audit.sh (opens in a new tab) checks the result, changes nothing, and prints PASS or FAIL for each setting. Each check is a command whose exit status is the answer:

fails=0
check() {
	local name=$1; shift
	if "$@" > /dev/null 2>&1; then
		echo "PASS  $name"
	else
		echo "FAIL  $name"
		fails=$((fails + 1))
	fi
}

shift drops the first argument, the name, so "$@" is the command to run (Lesson 12). The checks read like the track's table of contents:

check "password logins off" sshd_says "passwordauthentication no"
check "authorized_keys is 600 sam:sam" mode_is /home/sam/.ssh/authorized_keys "600 sam:sam"
check "sudo group needs a password" grep -qx 'Defaults !exempt_group' /etc/sudoers.d/10-bookshop
check "only SSH and web open to the network" only_public_ports

only_public_ports reads ss -tln (Lesson 9), skips anything listening only on 127.0.0.1, and fails if any port but 22 and 80 is left. Run it on the new server:

vagrant ssh -c 'sudo /setup/audit.sh'; echo "audit gave $?"

== SSH
PASS  password logins off
PASS  root logins off
PASS  ~sam/.ssh is 700 sam:sam
PASS  authorized_keys is 600 sam:sam
PASS  /home/sam not writable by others
== sudo
PASS  sudo group needs a password
PASS  all sudo rules check out
== Network
PASS  firewall active
PASS  only SSH and web open to the network
== Updates
PASS  automatic security updates on
== Services
PASS  nginx active
PASS  books-api active
PASS  postgresql active
PASS  books-backup.timer active
PASS  API answers 200
== Files
PASS  app settings 640 root:bookshop
PASS  data disk mounted
PASS  no world-writable files in /etc

All checks passed.
audit gave 0

Nineteen checks, and the exit status says the same thing as the last line: 0 all good, 1 something failed. That's what lets a scheduler or monitoring tool run it, and raise the alarm on a 1.

And from outside, the way an attacker would see it. Your computer, against the new server's address (vagrant ssh -c 'hostname -I' prints it):

ssh -o StrictHostKeyChecking=accept-new -i ~/.ssh/bookshop_ed25519 sam@192.168.34.132 'whoami; hostname; sudo -n true 2>&1'
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password -o NumberOfPasswordPrompts=0 sam@192.168.34.132
curl -s -o /dev/null -w "site from the Mac: %{http_code}\n" http://192.168.34.132/

Warning: Permanently added '192.168.34.132' (ED25519) to the list of known hosts.
sam
bookshop2
sudo: a password is required
sam@192.168.34.132: Permission denied (publickey).
site from the Mac: 200

Sam's key gets in, a password gets nowhere, sudo won't run without one, and the site answers. accept-new trusts a server it's never seen, but still refuses one whose key has changed: fine for a practice server you just built. For a real one, compare the fingerprint, as in Lesson 11. Sam has no password yet, so sudo can't be used at all until you set one: vagrant ssh, then sudo passwd sam.

Drift, and putting it back

Servers drift: someone fixes something by hand at 5pm, and the server no longer matches its recipe. Make it drift:

vagrant ssh -c 'sudo chmod 666 /etc/books-api/override.conf'
vagrant ssh -c 'sudo /setup/audit.sh'; echo "audit gave $?"

The Files part of the audit:

== Files
FAIL  app settings 640 root:bookshop
PASS  data disk mounted
FAIL  no world-writable files in /etc

4 check(s) failed.
audit gave 1

One chmod, two failures: wrong mode, and now anyone on the server can rewrite the app's settings. (Four in total: the other two were the audit's own bug, coming up next.) Put it back by running the recipe:

vagrant provision 2>&1 | grep -E "==>|installed|restarted"

==> default: Running provisioner: shell...
    default: ==> Packages (Lesson 8)
    default:     nginx, postgresql, unattended-upgrades and shellcheck installed; system up to date
    default: ==> Automatic security updates (Lesson 8)
    default: ==> Users (Lessons 5 and 11)
    default: ==> sudo needs a password (Lesson 11)
    default: ==> SSH: keys only, no root (Lesson 11)
    default: ==> Firewall: SSH and web only (Lesson 9)
    default: ==> Data disk at /srv/books-data (Lesson 10)
    default: ==> Database (Lessons 8 and 13)
    default: ==> The app (Lessons 7 and 9)
    default:     installed: /etc/books-api/override.conf
    default:     books-api restarted
    default: ==> Web server (Lessons 8 and 14)
    default: ==> Nightly backup (Lesson 13)
    default: ==> Done

Exactly one file put back, and one service restarted. Nothing else touched. The audit's Files section is back to three PASSes. And the ultimate repair is always there: vagrant destroy -f, then vagrant up, a brand-new server from the recipe, in minutes.

Try which changes each one catches, and fixes:

A fresh server, straight from setup.sh. Change something, then run the audit.

The audit and the setup script do different jobs. The setup script only puts back what it manages. Something it doesn't know about, like a test server someone started on port 8000, stays running. The audit checks the server as a whole, so it catches that too.

Three bugs, and what caught them

Building this lesson, the audit failed on a server that was meant to be perfect, and a review found a third problem before it could bite. All three were in my scripts, and all three are lessons from earlier in the track.

1. The setup script hid a failure. The first version of the SSH step was:

if put /tmp/10-bookshop.conf /etc/ssh/sshd_config.d/10-bookshop.conf 644 root:root; then
	sshd -t && systemctl reload ssh && echo "    sshd reloaded"
fi

And the first build said:

    default: ==> SSH: keys only, no root (Lesson 11)
    default:     installed: /etc/ssh/sshd_config.d/10-bookshop.conf
    default: Missing privilege separation directory: /run/sshd

sshd -t failed: on Ubuntu 24.04, SSH can start on demand, and the folder it needs didn't exist yet on a brand-new server. So the new settings were never applied, and the script carried on, and finished as if nothing had happened. Lesson 13's caveat: set -e doesn't stop for a failure on the left of &&. The fixed version puts each command on its own line, where set -e does apply, makes the folder first, and uses a reload that also works if SSH isn't running yet:

	mkdir -p /run/sshd
	sshd -t
	systemctl try-reload-or-restart ssh
	echo "    sshd settings tested and applied"

2. The audit lied. On that server, the audit said FAIL password logins off, yet sshd -T printed passwordauthentication no, and a password login really was refused. The check was:

sshd_says() { sshd -T | grep -qx "$1"; }

grep -q stops reading the moment it finds a match. sshd -T, still writing, gets cut off with SIGPIPE, the signal for "nobody's reading this any more", and exits with an error. With pipefail (Lesson 13), that counts as the whole pipe failing. Whether it happens depends on timing, which is why "root logins off" passed once and failed later. The fix is to let grep read everything:

sshd_says() { sshd -T 2>/dev/null | grep -x "$1" > /dev/null; }

3. put could report a failure as success. This one never happened on the server: a review of the script spotted it. The first version of put ended:

	install -m "$mode" -o "${owner%:*}" -g "${owner#*:}" "$src" "$dest"
	echo "    installed: $dest"
	return 0

put is always called inside a condition, and inside a condition, set -e is off for everything the function runs, not just the call. So if install failed, the next line ran anyway. On a test copy, with a source file that doesn't exist, the old version and the new one:

== old
  install: cannot stat 'missing': No such file or directory
      installed: /tmp/tmp.4FVc1s7osR/a
    -> reported changed
    script carried on
== new
  setup.sh: put: can't read missing

The old one printed the error, then installed, and returned 0: the caller restarted the app over a file that was never copied. The new one checks the source first, and die ends the whole script: exit works even where set -e doesn't. This is the same trap as bug 1, one level deeper, and why the ops toolkit in Lesson 13 checks every step by hand.

The lessons from all three: an audit is only as good as its checks, so check the checks; "it said it did it" isn't proof; and a function called in a condition has no set -e. After the first two fixes, the full rebuild above passed all nineteen. After the third, so did a vagrant provision, with every file unchanged.

Spot the problem

Work out what's wrong before you read the explanation.

The recipe that grew the server. A setup script contains echo "127.0.0.1 db.internal" >> /etc/hosts and echo "export APP_ENV=prod" >> /etc/profile. It's run nightly, to repair drift. A month later, /etc/hosts has thirty identical db.internal lines. What's wrong? It isn't idempotent: >> adds the line every run. It worked the first time, which is why nobody noticed. Ask first: grep -q "db.internal" /etc/hosts || echo ... >> /etc/hosts. The test for any setup script: run it twice, and the second run should change nothing.

Where you are now

That's the track. You started by asking a container what it was running. You can now find your way around any Linux server, read and search what it's saying, handle permissions, processes and services, install software, follow traffic through the network, manage disks and memory, log in with keys, write scripts that fail safely, schedule them, troubleshoot with a method, and build a locked-down server from nothing with one command, then prove it.

Next: the Ansible track turns this recipe into playbooks that run across many servers, and the AWS track puts servers like this in the cloud. Keep bookshop2 around for both: vagrant halt stops it, vagrant up brings it back, and vagrant destroy -f and vagrant up rebuild it from the recipe whenever you want a clean start.

Check yourself

Key terms

• Snowflake server: one built by hand, that nobody could recreate exactly.
• Setup script, or provisioning: the recipe that builds a server from nothing.
• Idempotent: doing it twice has the same effect as doing it once.
• Drift: a server slowly changing away from its recipe, through changes made by hand.
• Audit: a check of the server's real state against what it should be, changing nothing.
• cloud-init: the tool that runs a setup script when a cloud server first boots.
• SIGPIPE: the signal a program gets when nothing is reading what it writes.

Go deeper

• Add a check to audit.sh, for example that /srv/books-data/backups is mode 700 once a backup has run, and make it fail on purpose first.
• Run the audit on the first bookshop server: lp, then vagrant ssh -c 'sudo /practice/fresh/audit.sh'. Built by hand, does it pass?
• vagrant destroy -f && vagrant up in fresh, and time it. That's how long the bookshop takes to rebuild from nothing.
• Lesson 13, Shell scripts and cron, done properly, for set -e's exceptions, and Lesson 11, SSH and keys, for everything the SSH checks test.