Patching a fleet, one server at a time

By the end you'll know

Why servers are patched a few at a time, not all at once
How serial runs a play in batches, and max_fail_percentage stops it
How to upgrade packages with the apt module, and reboot only when it's needed
How the reboot module waits for a server to come back
How to reuse a health check as a role, so it can stop a rollout
Why a broken config can wait quietly until the next reboot

Before you start

You need: Lessons 1 to 8 done, with the three servers running and site.yml applied, and .vault_pass in place from Lesson 8.
Get the new files: in ansible-practice, git pull. You'll have patch.yml, two new roles, patch and health, and a three-line health.yml that uses the health role.
Allow time: the first patch run upgrades a couple of hundred packages and reboots each server. About five minutes, more on a slow connection.
Where commands run: your own computer's terminal, in ansible-practice.

The short version

The job. Every server had 238 package updates waiting. Upgrading them all at once is quick, until an update breaks something: then every server breaks together, and the site goes down.
The fix. patch.yml upgrades one web server at a time. Each one is patched, rebooted if it needs it, and health-checked before the next one starts.
What happened. web1 had a broken config, left on disk. The upgrade went fine, but after the reboot nginx couldn't start, and the health check failed. The run stopped there: web2 kept serving the site, and db1 was never touched.

Why patching needs care

Every server needs its packages updated regularly: security fixes, bug fixes, a new kernel now and then. On the bookshop's servers, there was plenty waiting:

BASH

ansible bookshop -m shell -a "apt list --upgradable 2>/dev/null | tail -n +2 | wc -l" -o
ansible bookshop -m command -a "uptime -s" -o

OUTPUT

web1 | CHANGED | rc=0 | (stdout) 238
web2 | CHANGED | rc=0 | (stdout) 238
db1 | CHANGED | rc=0 | (stdout) 238

web2 | CHANGED | rc=0 | (stdout) 2026-11-02 07:25:29
web1 | CHANGED | rc=0 | (stdout) 2026-11-02 07:24:42
db1 | CHANGED | rc=0 | (stdout) 2026-11-02 07:27:52

238 packages on each, counted by apt list --upgradable (the tail drops its heading line). And uptime -s says when each one last started: keep an eye on those times.

Ansible could upgrade all three at once, in parallel, as it's done everything so far. That's fast, and fine until one of the updates breaks something. Then it breaks on every server at the same moment, and the bookshop is down. Think of replacing the tyres on a car you're driving: one wheel at a time, and check it's tight before you start the next. The two web servers are there so one can serve the site while the other is being worked on.

Two roles

The work is split into two roles, so that each can be used on its own. patch brings one server up to date:

YAML

# roles/patch/tasks/main.yml
- name: Refresh the package list
  ansible.builtin.apt:
    update_cache: true
  changed_when: false

- name: List what's waiting to be upgraded
  ansible.builtin.command: apt list --upgradable
  register: upgradable
  changed_when: false

- name: Say how many
  ansible.builtin.debug:
    msg: "{{ upgradable.stdout_lines | length - 1 }} packages to upgrade on {{ inventory_hostname }}"

- name: Upgrade the packages
  ansible.builtin.apt:
    upgrade: safe

- name: Look for the reboot-required flag
  ansible.builtin.stat:
    path: /var/run/reboot-required
  register: reboot_flag

- name: Reboot, and wait for the server to come back
  ansible.builtin.reboot:
    reboot_timeout: 300
  when: reboot_flag.stat.exists

upgrade: safe upgrades everything it can without removing packages. The cautious choice for routine patching.
/var/run/reboot-required is a file Ubuntu creates when an update, usually a new kernel, only takes effect after a restart. No file, no reboot.
ansible.builtin.reboot restarts the server, then waits: it keeps trying to connect until the server is back, up to reboot_timeout seconds, and only then lets the play carry on. Without that, the next task would try to run on a server that's halfway through starting.

health is Lesson 6's health.yml, moved into a role, so patch.yml can run the same checks. health.yml itself is now three lines that use it:

YAML

- name: Check the web servers
  hosts: web
  roles:
    - health

One change on the way, which Lesson 6 promised: its rescue used to report a broken config and carry on, which hid the failure. Now, after reporting it, the rescue fails, so a broken server can stop a rollout:

YAML

  rescue:
    - name: Report what's wrong
      ansible.builtin.debug:
        msg: "nginx's config is broken on {{ inventory_hostname }}: {{ ansible_failed_result.stderr_lines[-2] | default('see above') }}"
    - name: Fail the run, now the reason is on screen
      ansible.builtin.fail:
        msg: "{{ inventory_hostname }} is not healthy: its nginx config is broken"

One at a time: serial

YAML

# patch.yml
- name: Patch the web servers, one at a time
  hosts: web
  become: true
  serial: 1
  max_fail_percentage: 0
  roles:
    - patch
    - health

- name: Patch the database server
  hosts: db
  become: true
  roles:
    - patch

serial: 1 runs the whole play on one server, start to finish, before the next one begins. Normally Ansible runs each task on every server, then moves to the next task. With serial, it works in batches: here, batches of one.
max_fail_percentage: 0 stops the play as soon as any server in a batch fails. More on why it's there in Spot the problem.
patch, then health: each web server is patched, rebooted if needed, and checked. Only a healthy server lets the next one start.
db1 is last, in its own play. It has no web health check, and only one of it, so there's no one-at-a-time to arrange.

The first run: a broken server

Before running it, web1 was given a problem: the same bad line Lesson 6 used, added to its nginx config by hand.

BASH

ansible web1 -m shell -a "echo bad_directive_here\; >> /etc/nginx/sites-available/bookshop" --become
time ansible-playbook patch.yml

OUTPUT

PLAY [Patch the web servers, one at a time] ************************************

TASK [Gathering Facts] *********************************************************
ok: [web1]

TASK [patch : Refresh the package list] ****************************************
ok: [web1]

TASK [patch : List what's waiting to be upgraded] ******************************
ok: [web1]

TASK [patch : Say how many] ****************************************************
ok: [web1] => {
    "msg": "238 packages to upgrade on web1"
}

TASK [patch : Upgrade the packages] ********************************************
changed: [web1]

TASK [patch : Look for the reboot-required flag] *******************************
ok: [web1]

TASK [patch : Reboot, and wait for the server to come back] ********************
changed: [web1]

TASK [health : Ask nginx for the home page, retrying while it starts] **********
FAILED - RETRYING: [web1]: Ask nginx for the home page, retrying while it starts (5 retries left).
FAILED - RETRYING: [web1]: Ask nginx for the home page, retrying while it starts (4 retries left).
FAILED - RETRYING: [web1]: Ask nginx for the home page, retrying while it starts (3 retries left).
FAILED - RETRYING: [web1]: Ask nginx for the home page, retrying while it starts (2 retries left).
FAILED - RETRYING: [web1]: Ask nginx for the home page, retrying while it starts (1 retries left).
fatal: [web1]: FAILED! => {"attempts": 5, "changed": false, "content": "", "elapsed": 0, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://localhost:80"}

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
web1                       : ok=7    changed=2    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   


real	1m25.505s

Read it from the top:

Only web1 ran. serial: 1 put it in a batch on its own.
238 packages upgraded, and the reboot flag was there, so web1 restarted. The reboot task waited for it to come back.
Then nginx didn't answer. "Connection refused", five times: nothing was listening on port 80 at all.
NO MORE HOSTS LEFT, and the run ended. web1 failed, the batch was 100% failed, so the play stopped. And because the run stopped, the database play never started either: the recap has only web1.

Why did nginx stop answering? Lesson 5 warned about exactly this. A broken config can sit on disk unnoticed while nginx keeps running its last good one, and the problem only appears when nginx has to start from scratch. A reboot is when that happens. The upgrade was fine; the reboot exposed a problem that was already there. Which is why the health check runs after the reboot, not before.

And what about the others?

OUTPUT

web1 | CHANGED | rc=0 | (stdout) 0
db1 | CHANGED | rc=0 | (stdout) 238
web2 | CHANGED | rc=0 | (stdout) 238

web2 | CHANGED | rc=0 | (stdout) 2026-11-02 07:25:29
db1 | CHANGED | rc=0 | (stdout) 2026-11-02 07:27:52
web1 | CHANGED | rc=0 | (stdout) 2026-11-03 03:52:37

web1: 0 packages waiting, and a new start time. web2 and db1: still 238 waiting, still the same start times. Nobody touched them. The bookshop stayed up the whole time, on web2. That's the point of one at a time: when something goes wrong, it goes wrong on one server.

Note what Ansible did not do: it didn't undo web1's upgrade. There's no rollback. It stopped, so that a person can look, and the damage stays at one server.

Fixing it, and patching the rest

web1's real problem was its config, and site.yml puts that right, as it did in Lesson 6:

BASH

ansible-playbook site.yml --limit web1
time ansible-playbook patch.yml

OUTPUT

PLAY RECAP *********************************************************************
web1                       : ok=16   changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

PLAY [Patch the web servers, one at a time] ************************************
...
TASK [patch : Say how many] ****************************************************
ok: [web1] => {
    "msg": "0 packages to upgrade on web1"
}
...
TASK [patch : Reboot, and wait for the server to come back] ********************
skipping: [web1]

TASK [health : Ask nginx for the home page, retrying while it starts] **********
ok: [web1]
...
PLAY [Patch the web servers, one at a time] ************************************
...
TASK [patch : Say how many] ****************************************************
ok: [web2] => {
    "msg": "238 packages to upgrade on web2"
}

TASK [patch : Upgrade the packages] ********************************************
changed: [web2]
...
TASK [patch : Reboot, and wait for the server to come back] ********************
changed: [web2]

TASK [health : Ask nginx for the home page, retrying while it starts] **********
ok: [web2]
...
PLAY [Patch the database server] ***********************************************
...
TASK [patch : Say how many] ****************************************************
ok: [db1] => {
    "msg": "238 packages to upgrade on db1"
}
...
TASK [patch : Reboot, and wait for the server to come back] ********************
changed: [db1]

PLAY RECAP *********************************************************************
db1                        : ok=7    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web1                       : ok=13   changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
web2                       : ok=14   changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   


real	2m53.050s

(Shortened.) site.yml made three changes on web1: the config rewritten without the bad line, nginx started, and the reload handler. Then patch.yml again:

web1 first, and already done: 0 packages, no reboot (skipping), healthy. Running it twice is safe.
Then a second PLAY [Patch the web servers...] header: that's web2's batch. Upgraded, rebooted, checked, healthy.
Then db1, upgraded and rebooted.

OUTPUT

db1 | CHANGED | rc=0 | (stdout) 0
web1 | CHANGED | rc=0 | (stdout) 0
web2 | CHANGED | rc=0 | (stdout) 0

web2 | CHANGED | rc=0 | (stdout) 2026-11-03 03:54:25
web1 | CHANGED | rc=0 | (stdout) 2026-11-03 03:52:36
db1 | CHANGED | rc=0 | (stdout) 2026-11-03 03:55:58

Nothing waiting anywhere, and three new start times, a couple of minutes apart: web1, then web2, then db1. At no point were both web servers down. And afterwards, site.yml and health.yml both came back clean:

OUTPUT

PLAY RECAP *********************************************************************
db1                        : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web1                       : ok=15   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web2                       : ok=15   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

PLAY RECAP *********************************************************************
web1                       : ok=8    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
web2                       : ok=8    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

The upgrades changed nothing Ansible manages. The servers are patched, and still exactly as described.

In real systems

Take the server out of traffic first. The bookshop has no load balancer, so web1 was simply down while it rebooted. In a real setup, a pre_tasks section removes the server from the load balancer before patching, and a post_tasks section puts it back after the health check passes. Visitors never reach a server that's restarting.
Bigger fleets use bigger batches. serial: [1, 5, "25%"] does one server first, as a canary, then five, then a quarter at a time. If the first one breaks, only one did.
Kubernetes does the same with nodes. Upgrading a cluster's nodes is this lesson again: cordon a node so no new pods land on it, drain the pods off it, upgrade and reboot it, check it, uncordon it, then the next. Managed Kubernetes services do it for you, one node at a time.
Patch on a schedule, not when you remember. Most teams patch weekly or monthly, at a quiet time, with this kind of playbook run from a scheduler or a pipeline. Ubuntu's unattended-upgrades can install security fixes automatically between runs, but it doesn't reboot one server at a time or check health afterwards.

Spot the problem

Work out what's wrong before you read the explanation.

Half the fleet, all at once. A team has ten web servers, and patches them with serial: "50%", five at a time, with no max_fail_percentage. A new package breaks their app. In the first batch, four servers fail their health check and one passes. Ansible carries on to the second batch, and patches the other five. Now nine of ten are broken. Why didn't it stop? By default, a batch only stops the play if every server in it fails. One of five passed, so the batch counted as not completely failed, and the play went on. With serial: 1, as in patch.yml, a batch is one server, so one failure is the whole batch, and the default would have stopped too; max_fail_percentage: 0 makes the rule explicit, and keeps it working if someone makes the batches bigger. For this team, max_fail_percentage: 0 would have stopped after the first batch, with five broken instead of nine. A one-server first batch, serial: [1, "50%"], would have stopped at one.

Keep this

To	Write or run
Run a play one server at a time	`serial: 1`, on the play
Grow the batches as it goes	`serial: [1, 5, "25%"]`
Stop at the first failure in a batch	`max_fail_percentage: 0`
Refresh the package list	`ansible.builtin.apt` with `update_cache: true`
Upgrade packages, cautiously	`ansible.builtin.apt` with `upgrade: safe`
See whether Ubuntu wants a reboot	`stat` on `/var/run/reboot-required`
Reboot, and wait for it	`ansible.builtin.reboot` with `reboot_timeout`
Fail on purpose, with a message	`ansible.builtin.fail` with `msg`
Count waiting updates	`apt list --upgradable`
When did a server last start?	`uptime -s`
Tasks before and after a play's roles	`pre_tasks:`, `post_tasks:`

Where you are now

The bookshop's servers are patched, by a playbook that does it one web server at a time, waits through each reboot, and won't move on from a server that isn't healthy. You've seen it stop on a broken server and keep the site up on the other. Lesson 10 makes sure the playbooks themselves are sound before they ever touch a server: checking them with a linter, and testing that each role does what it says.

Check yourself

Why patch the web servers one at a time?
What does serial: 1 change about how the play runs?
How does patch.yml decide whether to reboot?
web1's upgrade went fine. Why did its health check fail?
After web1 failed, what happened to web2 and db1?
With serial: "50%" and no max_fail_percentage, 4 of 5 servers fail. What happens?

Key terms

Patching: installing updates to a server's software, mostly security and bug fixes.
serial: a play setting that runs the play in batches of servers, one batch after another.
Batch: the group of servers a serial play is working on at one time.
max_fail_percentage: how many servers in a batch can fail before the play stops.
Canary: a first, small batch, so a bad change shows up on one server before it reaches the rest.
reboot-required: the file Ubuntu creates when an update needs a restart to take effect.
Rolling update: changing a fleet a few servers at a time, so it keeps serving throughout.

Go deeper

The Ansible documentation: Controlling playbook execution: strategies and more (opens in a new tab), which covers serial and max_fail_percentage, and the reboot module (opens in a new tab).
Try: break web2's config this time, the same way, and run health.yml. Which message do you get now that the rescue fails, and what's the recap?
Try: add a post_tasks: section to the web play in patch.yml, with a debug saying the server is back in service. Where does it run, compared with the roles?