Rebuilding the bookshop server

By the end you'll know

Why "can we rebuild it?" is the real test of configuration management
What a fresh server is missing, and the order to bring it back
Why an empty list of updates on a new server doesn't mean it's up to date
Why the inventory uses names, and what happens when an address changes
What happens to a change made by hand, when the server is rebuilt

Before you start

You need: Lessons 1 to 10 done, with the three servers running and ./test.sh passing.
No new files: this lesson uses the playbooks you already have. That's the point.
This deletes web2. vagrant destroy removes the VM and its disk. It's a practice server, and this lesson rebuilds it, but the delete is real.
Where commands run: your own computer's terminal, in ansible-practice.

The short version

The test. A server you can't rebuild is a server you can't afford to lose. So: destroy web2, and build it again from the playbooks alone.
The result. About three minutes from nothing to patched and healthy: a new VM, site.yml, then patch.yml. Same accounts, same config, same page, and test.sh passes.
What changed. A new IP address, which didn't matter, because the inventory uses names. And a fix someone had made by hand on web2 was gone, because it was never in the playbooks.

The server nobody can rebuild

Linux Lesson 15 started with a problem: a server built by hand, over weeks, that nobody could build again. When it dies, someone has to remember every package, every setting, every user. They won't remember all of it.

Everything in this track has been building the opposite: servers described in files, which Ansible can apply again and again. The real test of that is simple: delete a server, and build it again. If the files are complete, the new server is the same as the old one. If anything was done by hand, it's gone, and now you know. Think of a recipe you've only ever cooked from memory. The only way to know whether the written version is complete is to give it to someone else and see what comes out.

A fix made by hand

First, something realistic: a change made straight on the server, outside Ansible. Say someone needed bigger uploads, so they put a setting in nginx's conf.d folder on web2, and moved on:

BASH

ansible web2 -m copy -a "dest=/etc/nginx/conf.d/uploads.conf content=\"client_max_body_size 20m;\n\"" --become
ansible web -m command -a "ls /etc/nginx/conf.d" -o

OUTPUT

web2 | CHANGED | rc=0 | (stdout) uploads.conf
web1 | CHANGED | rc=0 | (stdout)

It's on web2, not web1, and not in any playbook. site.yml doesn't manage that folder, so it will never notice or remove it. Lesson 6's health check passes on both servers. And two things to compare afterwards:

BASH

ansible web2 -m command -a "cat /etc/bookshop-first-setup" -o
ansible web2 -m setup -a "filter=ansible_default_ipv4" | grep '"address"'

OUTPUT

web2 | CHANGED | rc=0 | (stdout) 2026-11-01T04:44:15Z
            "address": "192.168.34.134",

The first-setup time, from Lesson 3, and web2's IP address.

Destroying web2

BASH

time vagrant destroy -f web2
vagrant status

OUTPUT

==> web2: Stopping the VMware VM...
==> web2: Deleting the VM...

real	0m8.321s

web1                      running (vmware_desktop)
web2                      not created (vmware_desktop)
db1                       running (vmware_desktop)

Eight seconds, and web2 is gone: "not created", disk and all. Ansible notices straight away:

BASH

ansible web -m ping -o
ansible-playbook health.yml

OUTPUT

web2 | UNREACHABLE!: Failed to connect to the host via ssh: 
web1 | SUCCESS => {..., "ping": "pong"}

PLAY RECAP *********************************************************************
web1                       : ok=8    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
web2                       : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

unreachable=1, a new word in a recap: not failed, just not there. And web1 is still healthy. If this were a real outage, the bookshop would still be up, on one server, which is what the second web server is for.

A brand-new server

BASH

time vagrant up web2
vagrant ssh-config > ssh_config
ansible web2 -m ping -o

OUTPUT

==> web2: Machine booted and ready!
==> web2: Setting hostname...
==> web2: Configuring network adapters within the VM...

real	0m48.305s

web2 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3.12"},"changed": false,"ping": "pong"}

Under a minute for a new VM, from the same bento/ubuntu-24.04 box. vagrant ssh-config > ssh_config again, from Lesson 1: the new VM has a new SSH key, and possibly a new address, and this is how SSH, and so Ansible, finds out. Then a ping, and it answers. But it's an empty Ubuntu server:

BASH

ansible web2 -m command -a "ls /etc/nginx" -o
ansible web2 -m shell -a "apt list --upgradable 2>/dev/null | tail -n +2 | wc -l" -o

OUTPUT

web2 | FAILED! => {..., "stderr": "ls: cannot access '/etc/nginx': No such file or directory", ...}
web2 | CHANGED | rc=0 | (stdout) 0

No nginx at all. And 0 packages to upgrade, which looks like good news, and isn't. A brand-new server has never downloaded the list of available updates, so apt compares against an empty list and finds nothing newer. "0" here means "I don't know", not "up to date". Keep that in mind.

Rebuilding it

Two playbooks, in order: describe the server, then patch it.

BASH

time ansible-playbook site.yml --limit web2

OUTPUT

PLAY RECAP *********************************************************************
web2                       : ok=16   changed=12   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

real	0m22.689s

Twelve changes in 23 seconds: the packages, the login message, the staff accounts, the first-setup record, nginx and its config, the home page, the reload handler, and the app's settings with the vault-encrypted password. Everything the last ten lessons built, on a server that had none of it. Then:

BASH

time ansible-playbook patch.yml --limit web2

OUTPUT

TASK [patch : Say how many] ****************************************************
ok: [web2] => {
    "msg": "238 packages to upgrade on web2"
}

PLAY RECAP *********************************************************************
web2                       : ok=14   changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   

real	1m29.161s

238 packages, not 0. The first task in site.yml's baseline refreshed the package list, and now apt knows what's out there: the same 238 updates the other servers had in Lesson 9. The box the VM was built from is months old, so every rebuild starts out of date. That's why patching is part of rebuilding, and why patch.yml refreshes the list before counting. Upgraded, rebooted, health-checked: changed=2. (The --limit web2 also leaves db1 out, so the database play has nothing to do.)

Is it the same server?

BASH

ansible-playbook health.yml
ansible web2 -m command -a "getent passwd deploy sam" -o
ansible web2 -m command -a "getent passwd james" -o
ansible web2 -m command -a "ls -l /etc/bookshop/app.env" --become -o

OUTPUT

PLAY RECAP *********************************************************************
web1                       : ok=8    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
web2                       : ok=8    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   

web2 | CHANGED | rc=0 | (stdout) deploy:x:1001:1001:Deploys the website:/home/deploy:/bin/bash\nsam:x:1002:1002:Sam, owner:/home/sam:/bin/bash
web2 | FAILED! => {..., "cmd": ["getent","passwd","james"], ..., "rc": 2, ..., "stdout": "", ...}
web2 | CHANGED | rc=0 | (stdout) -rw------- 1 root root 132 Nov  3 12:45 /etc/bookshop/app.env

Healthy, by the same check that stops a rollout.
The same accounts, with the same user numbers as before: deploy 1001, sam 1002.
No James, and there never was one. His state: absent from Lesson 6 did nothing here, which is what it should do: an account that doesn't exist is already absent. That entry stays harmless on every server built from now on.
The app's settings, with the password from the vault, readable only by root.

And the page:

BASH

ansible web2 -m command -a "curl -s http://localhost:8080" -o

OUTPUT

web2 | CHANGED | rc=0 | (stdout) <h1>bookshop</h1>\n<p>Served by web2 (192.168.34.136), on port 8080.</p>\n<p>This server uses a non-standard port.</p>\n<p>Questions? ops@bookshop.example</p>

On port 8080, from the inventory, as before. But look at the address: 192.168.34.136. It was .134. The new VM got a new address from VMware, and nothing broke, because nothing used the old one. The inventory says web2; ssh_config turns that into an address; and the page fills it in from facts at run time. Names in the inventory, addresses looked up when needed. If inventory.ini had said 192.168.34.134, Ansible would now be knocking on an empty address.

Two things are different, as they should be:

OUTPUT

web2 | CHANGED | rc=0 | (stdout) 2026-11-03T12:45:31Z

web2 | CHANGED | rc=0 | (stdout) 
web1 | CHANGED | rc=0 | (stdout)

A new first-setup time, because it's a new server, and Lesson 3's creates: task ran once on it. And conf.d is empty on both servers. The hand-made uploads.conf is gone. Nobody removed it: it was on web2's disk, and web2's disk was deleted. It was never in the playbooks, so there was nothing to bring it back. If uploads over 1 MB had been working on web2, they've just stopped, and the only clue is a change someone made by hand and didn't write down.

Finally, the whole test, on all three servers:

BASH

time ./test.sh

OUTPUT

== idempotence: a second run must change nothing
PLAY RECAP *********************************************************************
db1                        : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web1                       : ok=15   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web2                       : ok=15   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
== health
ok
All checks passed.

real	0m30.871s

web2's recap matches web1's exactly: ok=15 changed=0. As far as Ansible can tell, they're the same server.

What it took

Step	Time
Destroy web2	8 seconds
A new VM	48 seconds
`site.yml`	23 seconds
`patch.yml`, with a reboot	1 minute 29 seconds
From nothing to patched and healthy	about 3 minutes

And what it didn't take: no notes, no memory, no logging in. The fix for the missing upload setting is the same as for everything else in this track: put it in the playbooks, in the nginx role, and it comes back with every rebuild.

In real systems

Rebuild on purpose, before you have to. Teams that rebuild servers regularly, from scratch, find the hand-made changes while the old server still exists to compare against. Teams that only rebuild in an emergency find them at the worst moment.
Hand changes happen; catch them. Someone will always fix something by hand during an incident. The habit that matters is writing it into the playbooks afterwards. Running site.yml with --check --diff on a schedule shows drift in the files Ansible manages, but not in files it doesn't, like uploads.conf. A rebuild finds both.
Cattle, not pets. A server you name, nurse and can't replace is a pet. A server you can rebuild in minutes, without a second thought, is cattle. Cloud autoscaling and Kubernetes take this all the way: servers and pods are replaced constantly, and anything not in an image, a playbook or a manifest is lost every time.
Start from a newer image. Every rebuild here started 238 updates behind. Teams that rebuild often build their own base image regularly, already patched, with a tool like Packer, so new servers start close to current.

Spot the problem

Work out what's wrong before you read the explanation.

The rebuild that worked, until Monday. A team's database server fails on a Friday night. They rebuild it from their playbooks in ten minutes, the health checks pass, and everyone goes home. On Monday, the finance team says the nightly sales report didn't arrive all weekend. The report was a cron job, added by hand on the old server two years ago by someone who's since left. It wasn't in the playbooks, so the rebuild didn't bring it back, and no health check looked for it. What would have caught it sooner? Rebuilding before the emergency, while the old server still existed: comparing the old and new servers' crontabs, packages and /etc would have shown the missing job. And the fix is the same as for uploads.conf: put the job in a role, with ansible.builtin.cron, so it's part of what the server is. A rebuild only brings back what's written down.

Keep this

To	Run
Delete a practice VM, disk and all	`vagrant destroy -f web2`
Make a new one from the same box	`vagrant up web2`
Tell SSH about the new key and address	`vagrant ssh-config > ssh_config`
Check Ansible can reach it	`ansible web2 -m ping`
Describe it	`ansible-playbook site.yml --limit web2`
Patch it	`ansible-playbook patch.yml --limit web2`
Check it	`ansible-playbook health.yml`, then `./test.sh`
See a server's address	`ansible web2 -m setup -a "filter=ansible_default_ipv4"`
Find hand-made files in a folder	`ansible web -m command -a "ls /etc/nginx/conf.d" -o`, and compare

Where you are now

You've destroyed a server and rebuilt it, patched and healthy, in about three minutes, from files alone. Everything the playbooks described came back, and the one thing they didn't describe was lost, which is the whole argument for writing it all down. Lesson 12 goes back to where this started: the bash script that built the Linux track's bookshop server, rewritten as a playbook, and checked by the same audit.

Check yourself

Why destroy and rebuild a server that's working fine?
The new web2 showed 0 packages to upgrade. What did that mean?
web2 came back as 192.168.34.136, not .134. Why did nothing break?
What happened to James's state: absent on the new web2?
Why was uploads.conf gone after the rebuild?

Key terms

Rebuild: creating a server again from nothing, from its description, rather than repairing it.
Drift: differences between what a server is and what its playbooks describe.
unreachable: a recap count for servers Ansible couldn't connect to at all, as opposed to failed.
Base image: the starting point a new server is built from, like the bento/ubuntu-24.04 box.
Cattle, not pets: treating servers as replaceable copies, rather than unique machines to be kept alive.
Packer: a tool for building base images, so new servers start already set up and patched.

Go deeper

Try: add uploads.conf to the nginx role, as a copy task with a client_max_body_size setting and notify: Reload nginx. Run site.yml, then destroy and rebuild web2 again. Is it there now?
Try: rebuild db1 the same way. What does site.yml do on it, and how long does it take?
Read: The Twelve-Factor App (opens in a new tab), on building software that can be thrown away and recreated. Its "Disposability" and "Config" sections say in software terms what this lesson did with servers.