Testing and linting playbooks

Testing before the servers

Every lesson so far has tested a playbook the same way: run it, and read the output. --check helped, but even that talks to the servers. It works, but it finds mistakes late, and some mistakes are only obvious to someone reading the code: a variable that could clash with another role's, a command that reports changed on every run.

Think of a building inspection. The plans are checked before anyone pours concrete, the wiring is checked before the walls go up, and the finished building is checked before anyone moves in. Each check is cheaper than finding the problem at the next stage. Playbooks have the same stages: check the text, check the habits, then check what it does to a server.

1. The syntax check

ansible-playbook --syntax-check site.yml patch.yml health.yml facts.yml

playbook: site.yml

playbook: patch.yml

playbook: health.yml

playbook: facts.yml

Each playbook's name, and nothing else: they all parse. --syntax-check reads the playbooks and the roles they use, checks the YAML and the Ansible keywords, and stops. It never connects to a server. It takes about a second, so there's no reason not to run it first. Here's what it catches, with roles: mistyped as role: in site.yml:

- name: Baseline for every bookshop server
  hosts: bookshop
  become: true
  role:
    - baseline

ERROR! 'role' is not a valid attribute for a Play

The error appears to be in '.../site.yml': line 5, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

A one-letter typo, found without touching a server. What it can't catch is anything that's valid but wrong: a misspelled variable name, a task that does the wrong thing. That needs the next two checks.

2. ansible-lint

ansible-lint reads playbooks and roles and reports habits that cause trouble: tasks without names, modules without their full names, commands that always report changed, and dozens more. Each finding names a rule, with a page explaining it. On the bookshop's repository:

ansible-lint

WARNING  Listing 8 violation(s) that are fatal

# Rule Violation Summary

  8 var-naming profile:basic tags:idiom

Failed: 8 failure(s), 0 warning(s) in 24 files processed of 30 encountered. Last profile that met the validation criteria was 'min'.

And the findings themselves, one per line with -f pep8:

roles/baseline/defaults/main.yml:12:1: var-naming[no-role-prefix]: Variables names from within roles should use baseline_ as a prefix. (vars: bookshop_staff)
roles/health/tasks/main.yml:2:13: var-naming[no-role-prefix]: Variables names from within roles should use health_ as a prefix. (register: page)
roles/health/tasks/main.yml:18:13: var-naming[no-role-prefix]: Variables names from within roles should use health_ as a prefix. (register: maintenance)
roles/health/tasks/main.yml:33:13: var-naming[no-role-prefix]: Variables names from within roles should use health_ as a prefix. (register: disk)
roles/nginx/defaults/main.yml:3:1: var-naming[no-role-prefix]: Variables names from within roles should use nginx_ as a prefix. (vars: http_port)
roles/nginx/defaults/main.yml:4:1: var-naming[no-role-prefix]: Variables names from within roles should use nginx_ as a prefix. (vars: site_name)
roles/patch/tasks/main.yml:7:13: var-naming[no-role-prefix]: Variables names from within roles should use patch_ as a prefix. (register: upgradable)
roles/patch/tasks/main.yml:20:13: var-naming[no-role-prefix]: Variables names from within roles should use patch_ as a prefix. (register: reboot_flag)

Everything else passed: every task has a name, every module its full name, every command its changed_when. All eight findings are one rule, the one Lesson 7 said was coming: a variable a role defines, or saves with register, should start with the role's name.

Why it matters. Ansible has one big pool of variables per server, shared by every role. If the nginx role and some other role both used http_port, or both registered page, one would quietly overwrite the other, and nothing would warn you. A prefix, nginx_http_port, health_page, means each role's names are its own.

The last line mentions profiles. ansible-lint has levels of strictness, from min up to production; the result says which level the code meets. min is the lowest.

(If you installed ansible-lint with Homebrew, it brings its own copy of Ansible, which can be newer than the one you run playbooks with. It may also print a warning about Python versions. Both are harmless here.)

Fixing the findings

Two kinds of fix. The register names only exist inside their role, so they're simply renamed, in the task and wherever the role reads them: page becomes health_page, reboot_flag becomes patch_reboot_flag, and so on.

The defaults are different, because the inventory sets them: http_port for web2, bookshop_staff in group_vars/all.yml. Renaming them everywhere would mean changing the inventory, health.yml, and every command in Lessons 4 to 9 that uses them. Instead, each role gets its own prefixed name, which reads the inventory's:

# roles/nginx/defaults/main.yml
# The port to listen on: the server's http_port if the inventory sets one
# (health.yml reads it too), otherwise 80.
nginx_http_port: "{{ http_port | default(80) }}"

# The site's name, for server_name and the log file. site_name still works
# as an override, as in Lesson 7.
nginx_site_name: "{{ site_name | default('bookshop') }}"

# roles/health/defaults/main.yml
# The port to check: the server's http_port, from the inventory. No fallback on
# purpose: a check that guesses the port could pass while checking the wrong one.
health_http_port: "{{ http_port }}"

The role's templates and tasks use the prefixed names. The inventory still describes the servers in its own words, http_port, and each role decides what to do with that. baseline gets baseline_staff: "{{ bookshop_staff | default([]) }}" the same way. One detail in the home page template: a value built with {{ }} is text, so the "non-standard port" test became {% if nginx_http_port | int != 80 %}, turning it back into a number before comparing.

ansible-lint

Passed: 0 failure(s), 0 warning(s) in 25 files processed of 32 encountered. Last profile that met the validation criteria was 'production'.

From min to production, the strictest profile. And since a rename is a refactor, Lesson 7's test applies:

ansible-playbook site.yml --check --diff
ansible-playbook site.yml
ansible-playbook site.yml --limit web1 --check --diff -e site_name=books

PLAY RECAP *********************************************************************
db1                        : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web1                       : ok=15   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web2                       : ok=15   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

-    server_name bookshop;
+    server_name books;

No diffs from the dry run, changed=0 from the real one, and Lesson 7's -e site_name=books still works. The servers can't tell anything changed.

3. Testing idempotence

The linter reads the code; it can't tell you what a run does. The most useful test of what a run does is one Lesson 3 started: run it twice, and the second run should change nothing. A playbook that changes something every time is either doing needless work, or fighting something else on the server. test.sh puts all the checks in one place:

#!/bin/bash
# Checks the playbooks, then runs site.yml twice: the second run must change nothing.
# Run with:  ./test.sh     Stops at the first problem, with a non-zero exit code.
set -euo pipefail
cd "$(dirname "$0")"

echo "== syntax"
ansible-playbook --syntax-check site.yml patch.yml health.yml >/dev/null
echo "ok"

echo "== lint"
ansible-lint --nocolor -q

echo "== apply"
ansible-playbook site.yml >/dev/null
echo "ok"

echo "== idempotence: a second run must change nothing"
recap=$(ansible-playbook site.yml | sed -n '/PLAY RECAP/,$p')
echo "$recap"
if echo "$recap" | grep -Eq 'changed=[1-9]|failed=[1-9]|unreachable=[1-9]'; then
  echo "FAILED: the second run changed something"; exit 1
fi

echo "== health"
ansible-playbook health.yml >/dev/null
echo "ok"
echo "All checks passed."

set -euo pipefail, from Linux Lesson 13, stops the script at the first command that fails. The cheap checks run first, so a typo fails in a second, not after a full run. The idempotence step keeps the second run's recap and fails if any server reports changed, failed or unreachable above zero.

time ./test.sh; echo "exit code: $?"

== syntax
ok
== lint
== apply
ok
== idempotence: a second run must change nothing
PLAY RECAP *********************************************************************
db1                        : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web1                       : ok=15   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web2                       : ok=15   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
== health
ok
All checks passed.

real	0m29.272s
exit code: 0

Exit code 0, in under 30 seconds. (The lint step printed nothing: -q keeps it quiet when there's nothing to say.)

A task that changes every time

Now a task that looks harmless, added to the end of the baseline role: record when Ansible last ran.

- name: Note when Ansible last ran
  ansible.builtin.shell: date > /etc/bookshop-last-run

The linter, first:

no-changed-when: Commands should not change things if nothing needs doing.
roles/baseline/tasks/main.yml:35 Task/Handler: Note when Ansible last ran

no-changed-when: a shell or command task with no changed_when, creates or removes, so it reports changed every time, whether or not anything needed doing. Suppose someone ignored that. With the lint step told to let it through, test.sh again:

== idempotence: a second run must change nothing
PLAY RECAP *********************************************************************
db1                        : ok=6    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web1                       : ok=16   changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web2                       : ok=16   changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
FAILED: the second run changed something
exit code: 1

changed=1 on every server, on the second run, and the test fails. Two different checks, one reading the code and one watching what it does, both caught the same task. That's the point of having both: the linter catches it before anything runs, and the test catches the cases a linter can't see. (The task was then removed, and test.sh passed again.)

Why does it matter if one harmless task always says changed? Because then changed stops meaning anything. If every run shows changes, nobody notices the run that changed something it shouldn't have. A recap of changed=0 is only useful when it's normal.

4. On every pull request

test.sh needs the servers, so it runs on your computer. But the syntax check and the linter only need the files, so GitHub can run them on every push. .github/workflows/lint.yml:

# Lint the playbooks on every push and pull request.
name: lint

on:
  push:
    branches: [main]
  pull_request:

jobs:
  ansible-lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install Ansible and ansible-lint
        run: pip install ansible-core ansible-lint

      # ansible.cfg expects the vault password in .vault_pass.
      # It comes from a repository secret, never from the repository itself.
      - name: Write the vault password
        run: printf '%s\n' "$VAULT_PASSWORD" > .vault_pass
        env:
          VAULT_PASSWORD: ${{ secrets.VAULT_PASSWORD }}

      - name: Syntax check
        run: ansible-playbook --syntax-check site.yml patch.yml health.yml

      - name: Lint
        run: ansible-lint

The shape is the CI/CD track's: check out the code, install the tools, run the checks. One new step: the vault password. ansible.cfg says to read it from .vault_pass, which git ignores, so the runner doesn't have one. It comes from a repository secret, set once from your computer:

gh secret set VAULT_PASSWORD --body "bookshop-practice"
gh secret list

✓ Set Actions secret VAULT_PASSWORD for Kwamib/ansible-practice
NAME            UPDATED               
VAULT_PASSWORD  less than a minute ago

GitHub keeps it encrypted, hides it in logs, and gives it to the workflow only while it runs. (The practice password is in the README anyway, but a real one would only ever live here.) Then a pull request, and a wait for the check:

gh pr create --fill --base main
gh pr checks --watch

https://github.com/Kwamib/ansible-practice/pull/6
All checks were successful
0 cancelled, 0 failing, 1 successful, 0 skipped, and 0 pending checks

   NAME                              DESCRIPTION  ELAPSED  URL
✓  lint/ansible-lint (pull_request)               16s      https://github.com/Kwamib/ansible-practice/actions/runs/41585849220/job/125751012814

16 seconds, passed. From now on, every pull request to this repository shows whether the playbooks pass, before anyone merges, and a task like the date one would turn the check red.

Spot the problem

Work out what's wrong before you read the explanation.

The red check that got merged. A team adds a lint workflow like this one. A month later, a pull request adds a shell task with no changed_when, and the check goes red. The author is in a hurry, the merge button is still green, and the pull request is merged with a note to fix it later. Nobody does. Soon every pull request is red, because main itself fails the linter, and people stop looking at the check at all. Three months later, a real problem is flagged in a pull request, and nobody notices, because the check is always red. What went wrong? The check was advice, not a rule. In the repository's branch protection settings, mark the check as required: then a pull request can't be merged while it's red. And fix, or deliberately skip with a written reason, every finding on main, so that red always means something new is wrong. It's the same idea as changed=0: a signal is only useful when its normal state is quiet.

Where you are now

The bookshop's playbooks are checked four ways: a syntax check, a linter that now passes its strictest profile, a test that runs everything twice and insists on changed=0, and a workflow that lints every pull request. The last lesson puts the whole track to the test it was built for: one of the servers is destroyed, and the playbooks build it again from nothing.

Check yourself

Key terms

• Syntax check: parsing playbooks and roles for errors, without connecting to a server.
• Linter: a tool that reads code and reports habits that cause trouble.
• ansible-lint: the linter for Ansible playbooks and roles.
• Rule: one thing a linter checks, like no-changed-when or var-naming[no-role-prefix].
• Profile: a level of strictness in ansible-lint, from min to production.
• Idempotence test: running a playbook twice and failing if the second run changes anything.
• Repository secret: a value stored by GitHub and given to workflows while they run, never shown in logs.
• Molecule: a tool that tests a role on a fresh server built for the test, then throws it away.

Go deeper

• The ansible-lint documentation: Rules (opens in a new tab), and Profiles (opens in a new tab).
• Try: rename a task to install nginx, starting with a lowercase letter, and run ansible-lint. Which rule fires, and does the profile change?
• Try: break a role on a branch, push it, and open a pull request. What does gh pr checks --watch show, and what does the pull request page say?