By the end you'll know
- How
loopruns one task over a list, and howloop_controlkeeps the output readable - Why removing something means saying so, not deleting the line
- How
whenruns a task only on the servers it applies to - How
register,untilandretrieswait for something to be ready - How
failed_whenandassertdecide what counts as a failure - What happens to the other servers when one fails, and how
blockandrescuerecover
Before you start
- You need: Lessons 1 to 5 done, with the three servers running and
site.ymlapplied. - Get the new files: in
ansible-practice,git pull. You'll have a newgroup_vars/all.ymlwith a staff list, a newhealth.yml, and a "Staff accounts" task insite.ymlin place of "Create the deploy user". - To follow along from the start:
group_vars/all.ymlarrives in its finished state, with james already marked as leaving. Delete its last line,state: absent, and you'll start where this lesson starts. - Where commands run: your own computer's terminal, in
ansible-practice.
One task, many accounts
Lesson 3 created one user, deploy, with one task. The bookshop has more people than that: Sam, the owner, and James, who works part-time. Three copies of the same task would work, until someone edits two of them and forgets the third. Instead, the people go in a list, in group_vars/all.yml, which applies to every server:
# Everyone who needs an account on the bookshop's servers.
# To remove someone, set their state to absent; deleting the line would leave the account behind.
bookshop_staff:
- name: deploy
comment: Deploys the website
- name: sam
comment: Sam, owner
- name: james
comment: James, part-time
A list of dictionaries, from Lesson 4: each person has a name and a comment. And one task in site.yml handles all of them:
- name: Staff accounts
ansible.builtin.user:
name: "{{ item.name }}"
comment: "{{ item.comment | default(omit) }}"
shell: /bin/bash
state: "{{ item.state | default('present') }}"
remove: "{{ item.state | default('present') == 'absent' }}"
loop: "{{ bookshop_staff }}"
loop_control:
label: "{{ item.name }}"
loopruns the task once for each thing in the list. Think of a mail merge: one letter, printed once per name on the list. Each time round, the current person isitem, soitem.nameis "deploy", then "sam", then "james".default(omit)means "if this person has no comment, leave the setting out altogether", rather than setting an empty one.stateispresentunless the list says otherwise. That's how someone leaves, below.removeis a comparison, so it'strueorfalse: when someone's account is removed, their home directory goes too.loop_control: labelcontrols what's printed for each item. Without it, Ansible prints the whole dictionary, comment and all, on every line.
The loop, run
ansible-playbook site.yml
TASK [Staff accounts] **********************************************************
changed: [db1] => (item=deploy)
changed: [web2] => (item=deploy)
changed: [web1] => (item=deploy)
changed: [db1] => (item=sam)
changed: [web1] => (item=sam)
changed: [web2] => (item=sam)
changed: [web1] => (item=james)
changed: [db1] => (item=james)
changed: [web2] => (item=james)
(Just the one task.) Nine results from one task: three people on three servers, each line labeled with the person. deploy already existed from Lesson 3, but reports changed here, because the task now gives it a comment, which it didn't have before. Did the accounts arrive?
ansible bookshop -m command -a "getent passwd deploy sam james" -o
web2 | CHANGED | rc=0 | (stdout) deploy:x:1001:1001:Deploys the website:/home/deploy:/bin/bash\nsam:x:1002:1002:Sam, owner:/home/sam:/bin/bash\njames:x:1003:1003:James, part-time:/home/james:/bin/bash
db1 | CHANGED | rc=0 | (stdout) deploy:x:1001:1001:Deploys the website:/home/deploy:/bin/bash\nsam:x:1002:1002:Sam, owner:/home/sam:/bin/bash\njames:x:1003:1003:James, part-time:/home/james:/bin/bash
web1 | CHANGED | rc=0 | (stdout) deploy:x:1001:1001:Deploys the website:/home/deploy:/bin/bash\nsam:x:1002:1002:Sam, owner:/home/sam:/bin/bash\njames:x:1003:1003:James, part-time:/home/james:/bin/bash
getent passwd looks up accounts, one line each: the name, the user number, the comment, the home directory and the shell. The same three accounts, with the same numbers, on every server. Run the playbook again, and every item is ok:
TASK [Staff accounts] **********************************************************
ok: [db1] => (item=deploy)
ok: [web2] => (item=deploy)
ok: [web1] => (item=deploy)
ok: [web2] => (item=sam)
...
ok: [web2] => (item=james)
A loop is still idempotent, item by item. Adding a fourth person is one more entry in the list, and the next run creates that account and leaves the others alone.
Someone leaves
James leaves the bookshop. The tempting fix is to delete him from the list. Don't: Ansible only manages what it's told about. With his entry gone, the loop would never mention him again, and his account would stay on every server, forgotten. Instead, say he's leaving. Add one line under his entry in group_vars/all.yml:
- name: james
comment: James, part-time
state: absent
ansible-playbook site.yml --diff
TASK [Staff accounts] **********************************************************
ok: [web1] => (item=deploy)
ok: [web2] => (item=deploy)
ok: [db1] => (item=deploy)
ok: [web2] => (item=sam)
ok: [web1] => (item=sam)
ok: [db1] => (item=sam)
changed: [db1] => (item=james)
changed: [web1] => (item=james)
changed: [web2] => (item=james)
Only James's items changed. (--diff shows nothing for this task: the user module doesn't report a before and after, only changed.) Is he gone?
ansible bookshop -m command -a "getent passwd james" -o
web2 | FAILED! => {..., "cmd": ["getent","passwd","james"], ..., "msg": "non-zero return code","rc": 2, ..., "stdout": "", ...}
web1 | FAILED! => {..., "rc": 2, ...}
db1 | FAILED! => {..., "rc": 2, ...}
(Shortened.) This FAILED is the answer you wanted. getent exits with code 2 when it can't find the name, and Ansible treats any non-zero exit as a failure. His account is gone from all three servers. Later, if there are hundreds of past staff in the list, you can tidy up entries once you're sure every server has run since; until then, the absent line is what removes them.
A health check
The rest of this lesson is a new playbook, health.yml. It checks the web servers and changes nothing, so it's safe to run at any time: after a deploy, from a schedule, or when someone says the site feels slow. It shows the other half of this lesson: tasks that run only sometimes, and failures handled on purpose.
- name: Check the web servers
hosts: web
tasks:
- name: Ask nginx for the home page, retrying while it starts
ansible.builtin.uri:
url: "http://localhost:{{ http_port }}"
return_content: true
register: page
retries: 5
delay: 2
until: page.status == 200
- name: The page should name this server
ansible.builtin.assert:
that:
- inventory_hostname in page.content
success_msg: "{{ inventory_hostname }} serves its own page"
fail_msg: "{{ inventory_hostname }}'s page doesn't mention it"
ansible.builtin.urimakes a web request, likecurl, from the server itself.http_portis each server's own, so web2 is asked on 8080.register: pagesaves the task's result in a variable, for later tasks: the status code, and withreturn_content, the page itself.until,retriesanddelay: if the condition isn't true, wait 2 seconds and try again, up to 5 more times. A service that's just been restarted can take a moment to answer; this is the difference between a check that waits and one that cries wolf.assertfails the task unless every condition underthatis true. Here: the page should contain the server's own name, which the Lesson 5 template puts there. A page served from the wrong server, or a stale one, fails the check.
Notice that until has no {{ }}. Conditions are already Jinja2, so you write the expression bare. The same goes for when and failed_when, next.
Tasks that run only sometimes
- name: Look for a maintenance page
ansible.builtin.stat:
path: /var/www/html/maintenance.html
register: maintenance
- name: Say so if there is one
ansible.builtin.debug:
msg: "{{ inventory_hostname }} has a maintenance page up"
when: maintenance.stat.exists
- name: Warn about small servers
ansible.builtin.debug:
msg: "{{ inventory_hostname }} has only {{ ansible_facts['memtotal_mb'] }} MB of memory"
when: ansible_facts['memtotal_mb'] < 2048
when runs a task only on the servers where its condition is true; everywhere else, the task is skipped. Two kinds of condition here:
- On something an earlier task found.
statlooks at a file without changing it, andregisterkeeps the answer.maintenance.stat.existsistrueorfalse. - On a fact, from Lesson 4.
memtotal_mbis the server's memory. The practice servers have 1 GB each, so this one will fire.
Deciding what counts as failure
- name: Check disk space, failing only above 90%
ansible.builtin.command: df --output=pcent /
register: disk
changed_when: false
failed_when: (disk.stdout_lines[-1] | trim | replace('%', '') | int) > 90
Normally a command task fails when the command exits non-zero. failed_when replaces that rule with your own. df --output=pcent / prints a heading and then a line like 21%. The condition takes the last line, stdout_lines[-1], then strips the spaces, removes the %, and turns what's left into a number. Above 90, the task fails. changed_when: false, from Lesson 3, because looking at the disk changes nothing.
A healthy run
ansible-playbook health.yml
TASK [Ask nginx for the home page, retrying while it starts] *******************
ok: [web1]
ok: [web2]
TASK [The page should name this server] ****************************************
ok: [web1] => {
"changed": false,
"msg": "web1 serves its own page"
}
ok: [web2] => {
"changed": false,
"msg": "web2 serves its own page"
}
TASK [Look for a maintenance page] *********************************************
ok: [web1]
ok: [web2]
TASK [Say so if there is one] **************************************************
skipping: [web1]
skipping: [web2]
TASK [Warn about small servers] ************************************************
ok: [web1] => {
"msg": "web1 has only 953 MB of memory"
}
ok: [web2] => {
"msg": "web2 has only 953 MB of memory"
}
TASK [Check disk space, failing only above 90%] ********************************
ok: [web2]
ok: [web1]
TASK [Test the config] *********************************************************
ok: [web1]
ok: [web2]
TASK [Note that this server was checked] ***************************************
ok: [web1] => {
"msg": "Checked web1, disk 21% full"
}
ok: [web2] => {
"msg": "Checked web2, disk 21% full"
}
PLAY RECAP *********************************************************************
web1 : ok=8 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
web2 : ok=8 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
(From the first task.) Both pages answered and named their own server. No maintenance page, so that task was skipping, counted as skipped=1. The memory warning fired, as expected: "1 GB" is 953 MB once the kernel has taken its share. And changed=0: a check that changes nothing, as promised. The last two tasks are the block at the end of the file, coming up below.
When one server fails
Now make it interesting. Put a maintenance page on web1, and stop nginx on web2:
ansible web1 -m copy -a "dest=/var/www/html/maintenance.html content=back_soon" --become
ansible web2 -m service -a "name=nginx state=stopped" --become
time ansible-playbook health.yml
TASK [Ask nginx for the home page, retrying while it starts] *******************
FAILED - RETRYING: [web2]: Ask nginx for the home page, retrying while it starts (5 retries left).
ok: [web1]
FAILED - RETRYING: [web2]: Ask nginx for the home page, retrying while it starts (4 retries left).
FAILED - RETRYING: [web2]: Ask nginx for the home page, retrying while it starts (3 retries left).
FAILED - RETRYING: [web2]: Ask nginx for the home page, retrying while it starts (2 retries left).
FAILED - RETRYING: [web2]: Ask nginx for the home page, retrying while it starts (1 retries left).
fatal: [web2]: FAILED! => {"attempts": 5, "changed": false, "content": "", "elapsed": 0, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://localhost:8080"}
TASK [The page should name this server] ****************************************
ok: [web1] => {
"changed": false,
"msg": "web1 serves its own page"
}
TASK [Look for a maintenance page] *********************************************
ok: [web1]
TASK [Say so if there is one] **************************************************
ok: [web1] => {
"msg": "web1 has a maintenance page up"
}
...
PLAY RECAP *********************************************************************
web1 : ok=9 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
web2 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
real 0m15.548s
Three things happened:
- web2 was retried, two seconds apart, with a countdown, before Ansible gave up: "Connection refused", nothing listening on 8080. That's the 15 seconds. If nginx had been starting rather than stopped, one of those retries would have succeeded, and the run would have carried on.
- Once web2 failed, it dropped out of the play. Every later task ran on
web1alone. web2's recap:ok=1(gathering facts) andfailed=1. - web1 carried on, and this time its maintenance page task ran instead of skipping. One server failing doesn't stop the others. That's usually what you want on a fleet; Lesson 9 will want the opposite.
Recovering on purpose: block and rescue
The last part of health.yml checks nginx's config:
- name: Check nginx's config, and recover if it's broken
become: true
block:
- name: Test the config
ansible.builtin.command: nginx -t
changed_when: false
rescue:
- name: Report it, instead of failing the whole run
ansible.builtin.debug:
msg: "nginx's config is broken on {{ inventory_hostname }}: {{ ansible_failed_result.stderr_lines[-2] | default('see above') }}"
always:
- name: Note that this server was checked
ansible.builtin.debug:
msg: "Checked {{ inventory_hostname }}, disk {{ disk.stdout_lines[-1] | trim }} full"
A block groups tasks, and settings like become apply to all of them. With rescue and always, it works like try, catch and finally in a programming language:
block: the tasks to try.rescue: runs only if a task in the block fails.ansible_failed_resultholds the failed task's result, so the rescue can say what went wrong.always: runs either way.
Think of a fire drill plan. The day's work is the block. If the alarm goes, the rescue takes over: everyone out, by the plan. And whatever happens, someone signs the register at the end of the day.
Start web2's nginx again, and break web1's config with a line nginx doesn't know:
ansible web2 -m service -a "name=nginx state=started" --become
ansible web1 -m shell -a "echo bad_directive_here\; >> /etc/nginx/sites-available/bookshop" --become
ansible-playbook health.yml
TASK [Ask nginx for the home page, retrying while it starts] *******************
ok: [web2]
ok: [web1]
...
TASK [Test the config] *********************************************************
fatal: [web1]: FAILED! => {"changed": false, "cmd": ["nginx", "-t"], ..., "rc": 1, ..., "stderr_lines": ["2026/11/02 03:52:17 [emerg] 50777#50777: unknown directive \"bad_directive_here\" in /etc/nginx/sites-enabled/bookshop:17", "nginx: configuration file /etc/nginx/nginx.conf test failed"], ...}
ok: [web2]
TASK [Report it, instead of failing the whole run] *****************************
ok: [web1] => {
"msg": "nginx's config is broken on web1: 2026/11/02 03:52:17 [emerg] 50777#50777: unknown directive \"bad_directive_here\" in /etc/nginx/sites-enabled/bookshop:17"
}
TASK [Note that this server was checked] ***************************************
ok: [web1] => {
"msg": "Checked web1, disk 21% full"
}
ok: [web2] => {
"msg": "Checked web2, disk 21% full"
}
PLAY RECAP *********************************************************************
web1 : ok=9 changed=0 unreachable=0 failed=0 skipped=0 rescued=1 ignored=0
web2 : ok=8 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
nginx -tfailed on web1, with the exact problem: an unknown directive, on line 17 of the bookshop config. Thefatalline still prints.- The rescue caught it, and reported the important line.
stderr_lines[-2]is the second-to-last line of the error, where nginx names the problem;default('see above')covers an error with only one line. alwaysran on both servers, broken or not.- The recap says
failed=0andrescued=1. web1 didn't drop out: a rescued failure counts as handled. - The page check still passed on web1, like Lesson 5: nginx is running its last good config, and the broken one is waiting on disk for the next reload or reboot.
That last recap is worth a second look, because rescuing a failure hides it. The playbook ends with failed=0, and exits as a success. A person reading the output sees the rescue message; a scheduled job that only checks whether the run succeeded sees nothing wrong. For this check, that's the wrong choice, and Lesson 9 will want to know. Use rescue to clean up or to try another way, and when the problem still needs a person, end the rescue with ansible.builtin.fail so the run still fails, with your message.
Putting it back
Remove the maintenance page, and let site.yml repair web1's config:
ansible web1 -m file -a "path=/var/www/html/maintenance.html state=absent" --become
ansible-playbook site.yml
ansible-playbook health.yml
PLAY RECAP *********************************************************************
db1 : ok=5 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
web1 : ok=14 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
web2 : ok=13 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
PLAY RECAP *********************************************************************
web1 : ok=8 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
web2 : ok=8 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
web1's two changes are the template writing the config back without the bad line, and the handler reloading nginx: ok=14, one more than web2's 13, is that handler. Then the health check is back to where it started. The hand edit lasted exactly until the next run, as the message of the day on every server warns.
In real systems
- Lists drive most fleets. Users, packages, firewall rules, cron jobs, sites: a list in a variables file, and one looped task. The list is what people review and change; the task hardly ever changes.
- Removing is a change you write down.
state: absentis how accounts, packages and files leave. The same is true in Terraform (deleting a resource there does destroy it, because Terraform keeps a record of what it made; Ansible keeps no such record, so it can't notice what you stopped mentioning). - Waiting with
untilbeats a fixed sleep. Apauseof 30 seconds is too long when the service is quick and too short when it's slow. Retry on the thing you actually need, like the page answering. - Decide how a fleet fails. By default, a failed server drops out and the rest carry on.
any_errors_fatal: trueon a play stops everything at the first failure, andmax_fail_percentagestops once too many servers fail. Lesson 9 needs both ideas.
Spot the problem
Work out what's wrong before you read the explanation.
The check that always passes. Someone writes a task to check that a backup finished: ansible.builtin.command: grep -q "backup complete" /var/log/backup.log, with failed_when: false, "so the playbook doesn't stop if the log is missing". It's been green every night for months. Then a restore is needed, and there's no backup from the last six weeks. What happened? failed_when: false means this task never fails, whatever happens. The grep found nothing for six weeks, exited 1, and the task reported ok. The fix is to decide what failure means, not to switch it off: register the result and fail on it (failed_when: result.rc != 0), or use ignore_errors: true if you really must carry on, which at least prints the failure and counts it in the recap as ignored. Every check is worth breaking once on purpose, as this lesson did, to see it fail.
Keep this
| To | Write or run |
|---|---|
| Run a task once per item | loop: "{{ bookshop_staff }}", then item inside it |
| Print only part of each item | loop_control: with label: "{{ item.name }}" |
| Leave a setting out if there's no value | {{ item.comment | default(omit) }} |
| Remove something | state: absent, not deleting the line |
| Save a task's result | register: page |
| Run a task only sometimes | when: maintenance.stat.exists, no {{ }} |
| Retry until something is true | until:, retries: 5, delay: 2 |
| Fail unless a condition holds | ansible.builtin.assert with that: |
| Decide what failure means | failed_when: <condition> |
| Look at a file without changing it | ansible.builtin.stat, then .stat.exists |
| Make a web request | ansible.builtin.uri with url, return_content |
| Try, recover, and finish | block:, rescue:, always: |
| Fail on purpose, with a message | ansible.builtin.fail with msg |
| Stop everything at the first failure | any_errors_fatal: true, on the play |
| Look up accounts | getent passwd sam |
Where you are now
Your servers' accounts come from one list, and leaving is one line. You have a health check you can run any time: it waits for what it needs, runs tasks only where they apply, and decides for itself what counts as failure. And you've seen what a failure does to the rest of a run, and how to recover from one without hiding it. site.yml is getting long, with a baseline play, a web play and their files side by side. Lesson 7 splits it into roles: reusable pieces, each with its own tasks, templates, handlers and defaults.
Check yourself
Inside a task with loop, what is item?
James leaves. Why set state: absent instead of deleting his entry from the list?
Why does when: maintenance.stat.exists have no {{ }} around it?
web2's nginx was stopped. What did the other tasks do on web2 after the page check failed?
nginx -t failed on web1, and the rescue ran. What did the recap say?
A check has failed_when: false. What does it report when the thing it checks is broken?
Key terms
- loop: runs one task once for each entry in a list.
- item: the current entry, inside a looped task.
- loop_control: settings for a loop, like
label, which controls what's printed per item. - register: saves a task's result in a variable, for later tasks.
- when: a condition; the task runs only where it's true, and is skipped elsewhere.
- until, retries, delay: retry a task until a condition is true, a set number of times, with a pause.
- assert: a task that fails unless its conditions are true.
- failed_when: your own rule for when a task counts as failed.
- block, rescue, always: tasks to try, tasks to run if one fails, and tasks to run either way.
- ansible_failed_result: the failed task's result, available inside
rescue.
Go deeper
- The Ansible documentation: Loops (opens in a new tab), Conditionals (opens in a new tab), Error handling in playbooks (opens in a new tab), and Blocks (opens in a new tab).
- Try: add yourself to
bookshop_staffwithout acomment, and runsite.yml. Thengetent passwdyour name. What's in the comment field? - Try: add an
ansible.builtin.failtask to the end of the rescue, break web1's config again, and runhealth.yml. What does the recap say now, and what doesecho $?print straight afterwards?