From setup.sh to a playbook

Where this started

Linux Lesson 15 ended with setup.sh (opens in a new tab): everything the Linux track did by hand, as one script that's safe to run again. It was careful work. Every step checks before it changes anything, a helper function compares files before replacing them, and the audit that checked the result caught three bugs in the script itself.

This lesson asks a simple question: what does the same job look like in Ansible? Same server, same files, same order, same audit at the end. Think of the same recipe written two ways. One says "check whether the oven's at 200; if it isn't, turn the dial until it is". The other says "oven: 200". The second only works if the oven knows how to get itself to 200. Ansible's modules are ovens that know.

The new folder

The new server gets a folder of its own in ansible-practice, so it doesn't mix with web1, web2 and db1:

• bookshop/Vagrantfile: one server, bookshop3, with 2 GB of memory like the original. No provisioner: the playbook does the work.
• bookshop/inventory.ini and bookshop/ansible.cfg: one server, and the same settings as the top folder.
• bookshop/files/: the nine files setup.sh installs, copied from linux-practice: the app, its settings, its unit file, nginx's site, the backup script and timer, and Sam's SSH key.
• bookshop/audit.sh: the Linux track's audit, unchanged.
• bookshop/bookshop.yml: the playbook.

Collections

Everything so far has used modules named ansible.builtin: the ones that come with Ansible. setup.sh needs three things they don't cover: the ufw firewall, PostgreSQL, and mounting a disk. Those modules come in collections, add-on packs of modules from Ansible Galaxy (opens in a new tab). requirements.yml lists the ones this repository needs:

# Collections the playbooks use beyond ansible.builtin.
# Install with: ansible-galaxy collection install -r requirements.yml
collections:
  - name: community.general     # ufw
  - name: community.postgresql  # postgresql_db, postgresql_table, postgresql_query
  - name: ansible.posix         # mount

ansible-galaxy collection install -r requirements.yml
ansible-galaxy collection list | grep -E "^(community.general|community.postgresql|ansible.posix) "

Starting galaxy collection install process
Nothing to do. All requested collections are already installed. If you want to reinstall them, consider using `--force`.

ansible.posix                            1.6.2  
community.general                        10.7.1 
community.postgresql                     3.14.2 

They were already there: some ways of installing Ansible include them. If yours doesn't, the same command downloads them. Modules from a collection are named after it: community.general.ufw, community.postgresql.postgresql_db, ansible.posix.mount. The lint workflow from Lesson 10 got one new step, ansible-galaxy collection install -r requirements.yml, so GitHub has them too.

A fresh server

cd bookshop
time vagrant up
vagrant ssh-config > ssh_config
ansible bookshop3 -m ping -o

==> bookshop3: Machine booted and ready!
==> bookshop3: Setting hostname...
==> bookshop3: Configuring network adapters within the VM...

real	0m49.609s

bookshop3 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3.12"},"changed": false,"ping": "pong"}

A bare Ubuntu server, as in Lesson 11.

Step by step: script and playbook

The playbook follows setup.sh step by step, in the same order. Here are the steps where the two differ most.

Files. setup.sh has a helper, put, about 20 lines, that compares the new file with what's there, checks the mode and owner, and only replaces it if something's different:

put() {
	local src=$1 dest=$2 mode=$3 owner=$4
	...
	if [ -f "$dest" ] && cmp -s -- "$src" "$dest" \
		&& [ "$(stat -c %a -- "$dest")" = "$mode" ] \
		&& [ "$(stat -c %U:%G -- "$dest")" = "$owner" ]; then
		echo "    unchanged: $dest"
		return 1
	fi
	install -m "$mode" -o "${owner%:*}" -g "${owner#*:}" -- "$src" "$dest" \
		|| die "put: couldn't install $dest"
	...
}

In the playbook, that's what ansible.builtin.copy does already, every time:

    - name: Sam's SSH key
      ansible.builtin.copy:
        src: files/sam.pub
        dest: /home/sam/.ssh/authorized_keys
        owner: sam
        group: sam
        mode: "0600"

Restarting only after a change. setup.sh keeps a flag, app_changed=0, sets it to 1 whenever put replaces one of the app's files, and restarts the app only if it's 1. In the playbook, that's a handler, from Lesson 5: the copy task says notify: Restart books-api, and the handler runs once, at the end, only if something changed. Four files, one loop:

    - name: The app and its unit file
      ansible.builtin.copy:
        src: "files/{{ item.src }}"
        dest: "{{ item.dest }}"
        mode: "{{ item.mode }}"
        owner: root
        group: "{{ item.group | default('root') }}"
      loop:
        - { src: books-api, dest: /usr/local/bin/books-api, mode: "0755" }
        - { src: app.conf, dest: /etc/books-api/app.conf, mode: "0640", group: bookshop }
        - { src: override.conf, dest: /etc/books-api/override.conf, mode: "0640", group: bookshop }
        - { src: books-api.service, dest: /etc/systemd/system/books-api.service, mode: "0644" }
      loop_control:
        label: "{{ item.dest }}"
      notify: Restart books-api

Testing a file before it's used. setup.sh writes the sudo rule to /tmp, checks it with visudo -cqf, and only then copies it into place, because a broken sudo rule can lock everyone out of sudo. The playbook does the same with validate:, which Lesson 5 mentioned: Ansible writes the new file to a temporary path, runs the command on it with %s as the path, and only puts it in place if the command succeeds.

    - name: Make the sudo group type its password
      ansible.builtin.copy:
        dest: /etc/sudoers.d/10-bookshop
        content: "Defaults !exempt_group\n"
        mode: "0440"
        validate: visudo -cf %s

    - name: SSH with keys only, and no root
      ansible.builtin.copy:
        dest: /etc/ssh/sshd_config.d/10-bookshop.conf
        content: |
          PasswordAuthentication no
          PermitRootLogin no
        mode: "0644"
        validate: /usr/sbin/sshd -t -f %s
      notify: Reload ssh

The firewall. Three ufw commands in the script, three tasks with community.general.ufw here, and the same result. The difference is the report: the script always prints "ufw active", whether or not it changed anything; the module says ok or changed, like everything else.

The data disk. setup.sh checks for the image, makes it, checks /etc/fstab with grep, adds a line if it's missing, and mounts the disk if it isn't mounted. The playbook needs two tasks. creates: makes the image only once, and ansible.posix.mount with state: mounted does all the rest: the line in /etc/fstab, and mounting it now.

    - name: Make the disk image
      ansible.builtin.shell:
        cmd: fallocate -l 90M {{ data_image }} && mkfs.ext4 -q {{ data_image }}
        creates: "{{ data_image }}"

    - name: Mount it, now and at every boot
      ansible.posix.mount:
        path: /srv/books-data
        src: "{{ data_image }}"
        fstype: ext4
        opts: loop,nofail
        passno: 2
        state: mounted

The database. The script runs psql queries and compares their output to decide what to do. The playbook uses community.postgresql: postgresql_db for the database, postgresql_table for the table, each idempotent on its own. The three sample books only go in if the table is empty, with a count and a when:, from Lesson 6:

    - name: Count the books
      community.postgresql.postgresql_query:
        login_db: books
        query: SELECT count(*) AS n FROM books
      become: true
      become_user: postgres
      register: books_count
      changed_when: false

    - name: Add three books, if there are none
      community.postgresql.postgresql_query:
        login_db: books
        query: >-
          INSERT INTO books (title, stock) VALUES
          ('The Linux Command Line', 4), ('Designing Data-Intensive Applications', 2), ('The Phoenix Project', 7)
      become: true
      become_user: postgres
      when: books_count.query_result[0].n == 0

become_user: postgres runs a task as the postgres user, as the script's runuser -u postgres did. Two new packages make that work, installed with the others: acl, which lets Ansible hand its temporary files to another user, and python3-psycopg2, which lets the modules talk to PostgreSQL. The script needed neither. That's the one way the playbook asks more of the server.

Checking before running

ansible-playbook --syntax-check bookshop.yml
cd .. && ansible-lint

The syntax check passed. The linter found five things in all: one on the first run, and four more when it was run on the whole repository. The four were one rule:

partial-become[task]: ``become_user`` should have a corresponding ``become`` at the same level as itself.
bookshop/bookshop.yml:136 Task/Handler: The books database
...

The database tasks had become_user: postgres, and relied on the play's become: true, one level up. That works. But someone reading the task alone can't see it, and moving it into a play without become would quietly break it. The fix is the become: true you see on each database task above.

The fifth was about the handler that reloads SSH, which runs systemctl itself:

command-instead-of-module: systemctl used in place of systemd module
bookshop/bookshop.yml:255 Task/Handler: Reload ssh

Usually right, and here deliberately not. setup.sh noted why: on Ubuntu 24.04, ssh can be started on demand, so it may not be running, and a plain reload fails on a stopped service. systemctl try-reload-or-restart reloads it if it's running, and does nothing if not. So the command stays, and the rule is skipped on that one line, with the reason written next to it, as Lesson 10 described:

    # Not the systemd_service module, on purpose: on Ubuntu 24.04 ssh can be started
    # on demand, so it may not be running, and a plain reload fails on a stopped
    # service. try-reload-or-restart reloads it if it's running, and does nothing if not.
    - name: Reload ssh
      ansible.builtin.command: systemctl try-reload-or-restart ssh  # noqa: command-instead-of-module
      changed_when: true

Passed: 0 failure(s), 0 warning(s) in 28 files processed of 49 encountered. Last profile that met the validation criteria was 'production'.

The first run

time ansible-playbook bookshop.yml

PLAY [Build the bookshop server] ***********************************************

TASK [Install the packages] ****************************************************
changed: [bookshop3]

TASK [Bring everything up to date] *********************************************
changed: [bookshop3]

TASK [Turn on automatic security updates] **************************************
ok: [bookshop3]

TASK [The app's own user, which can't log in] **********************************
changed: [bookshop3]

TASK [Sam, who can use sudo] ***************************************************
changed: [bookshop3]

TASK [Sam's home, closed to others] ********************************************
ok: [bookshop3]
...
TASK [Make the sudo group type its password] ***********************************
changed: [bookshop3]
...
TASK [Turn the firewall on] ****************************************************
changed: [bookshop3]

TASK [Make the disk image] *****************************************************
changed: [bookshop3]

TASK [Mount it, now and at every boot] *****************************************
changed: [bookshop3]
...
TASK [The books database] ******************************************************
[WARNING]: Module remote_tmp /var/lib/postgresql/.ansible/tmp did not exist and
was created with a mode of 0700, this may cause issues when running as another
user. To avoid this, create the remote_tmp dir with the correct permissions
manually
changed: [bookshop3]
...
TASK [Add three books, if there are none] **************************************
changed: [bookshop3]
...
TASK [The app and its unit file] ***********************************************
changed: [bookshop3] => (item=/usr/local/bin/books-api)
changed: [bookshop3] => (item=/etc/books-api/app.conf)
changed: [bookshop3] => (item=/etc/books-api/override.conf)
changed: [bookshop3] => (item=/etc/systemd/system/books-api.service)
...
RUNNING HANDLER [Reload ssh] ***************************************************
changed: [bookshop3]

RUNNING HANDLER [Restart books-api] ********************************************
changed: [bookshop3]

RUNNING HANDLER [Reload nginx] *************************************************
changed: [bookshop3]

PLAY RECAP *********************************************************************
bookshop3                  : ok=37   changed=31   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

real	2m3.203s

(Shortened.) 31 changes in two minutes, most of it the package upgrade. The [WARNING] is the same harmless one Lesson 8 saw, this time for the postgres user. And two tasks came back ok on a brand-new server, which is worth a look:

• Automatic security updates: ok. Installing the unattended-upgrades package writes 20auto-upgrades itself, with exactly these two lines. The file was already right, so copy left it alone. setup.sh rewrote it anyway, and couldn't tell you it hadn't needed to.
• Sam's home, closed to others: ok. Ubuntu 24.04 creates new home folders as 750 already. Again, nothing to do, and the playbook says so.

The second run:

PLAY RECAP *********************************************************************
bookshop3                  : ok=33   changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   

real	0m27.394s

changed=0 in 27 seconds, and skipped=1: the three books weren't added again, because the count was 3. No handlers ran, because nothing notified them.

The audit

The real test: the Linux track's audit.sh, unchanged, which knows nothing about Ansible. ansible.builtin.script copies a script from your computer to the server and runs it there:

ansible bookshop3 -m script -a audit.sh --become

== SSH
PASS  password logins off
PASS  root logins off
PASS  ~sam/.ssh is 700 sam:sam
PASS  authorized_keys is 600 sam:sam
PASS  /home/sam not writable by others
== sudo
PASS  sudo group needs a password
PASS  all sudo rules check out
== Network
PASS  firewall active
PASS  only SSH and web open to the network
== Updates
PASS  automatic security updates on
== Services
PASS  nginx active
PASS  books-api active
PASS  postgresql active
PASS  books-backup.timer active
PASS  API answers 200
== Files
PASS  app settings 640 root:bookshop
PASS  data disk mounted
PASS  no world-writable files in /etc

All checks passed.

(The audit's lines, from the module's stdout.) All 18 checks, first time. And the app and its data:

ansible bookshop3 -m command -a "curl -s http://127.0.0.1/api/" -o
ansible bookshop3 -m command -a "psql -d books -tAc \"SELECT title, stock FROM books ORDER BY id\"" --become --become-user postgres

bookshop3 | CHANGED | rc=0 | (stdout) {"books": ["Dune", "Beloved", "Kindred"], "served_by": "bookshop3"}

bookshop3 | CHANGED | rc=0 >>
The Linux Command Line|4
Designing Data-Intensive Applications|2
The Phoenix Project|7

The same server setup.sh builds, built by a playbook, and checked by the script's own audit.

To be fair to setup.sh: the playbook had a head start. It was written after the script's three bugs had been found and fixed, and it borrowed the script's answers to the hard parts, like /run/sshd and try-reload-or-restart. Passing first time is partly the script's work.

Script and playbook, side by side

The playbook is longer. YAML is wordy: every task has a name, every setting its own line. But look at what's missing. The script spends its care on checking: is the file different, did the step fail, did anything change. Linux Lesson 15's three bugs were all in that checking code, and two were failures the script didn't notice. The playbook has no checking code to get wrong. Each module checks for itself, reports what it did, and fails loudly when it can't. What's left is a description of the server.

When bash is still the better choice. A first-boot script on a cloud server, with nothing installed yet. A one-off job on one machine. Anything where adding Python, SSH access and a control machine is more work than the job itself. And as Linux Lesson 15 showed, a careful script works. It just takes more care to keep it working.

Spot the problem

Work out what's wrong before you read the explanation.

The run that was always ok. Someone tidying bookshop.yml decides the count is clumsy. They delete "Count the books", remove the when: from "Add three books", and, because Lesson 10's test.sh now fails on its second run, add changed_when: false to the insert. test.sh passes, the recap says changed=0, and the change is merged. A month later, the shop's stock report lists every title dozens of times. What happened? changed_when: false doesn't stop a task from changing anything. It only stops it from saying so. Every run inserted three more books, and reported ok. The idempotence test was right to fail, and the "fix" switched off its evidence. changed_when is for describing what a task does, like nginx -t, which really changes nothing. To make a task idempotent, change what it does: the count and when:, or a unique constraint on title so the database itself refuses duplicates. And when a test fails, fix the task, not the test.

Where you are now

That's the track. You've gone from one ad-hoc ping to a fleet described in roles, with encrypted secrets, rolling patches, a linter and tests in GitHub, a server rebuilt in three minutes, and now the Linux track's whole server as a playbook that passes the same audit. When you're done, vagrant destroy -f in the bookshop folder deletes bookshop3, and vagrant up in ansible-practice brings web1, web2 and db1 back.

Check yourself

Key terms

• Collection: a pack of modules, plugins and roles, installed from Ansible Galaxy, like community.general.
• ansible.builtin: the modules that come with Ansible itself.
• requirements.yml: the list of collections and roles a project needs, for ansible-galaxy to install.
• validate: a command that tests a new file before copy or template puts it in place.
• become_user: the user a task runs as, once become is on: here, postgres.
• script module: copies a script from your computer to a server, and runs it there.
• Migration: moving from one way of doing a job to another, a step at a time, checking each step.

Go deeper

• The Linux track's Lesson 15, to read setup.sh and its three bugs side by side with bookshop.yml.
• The Ansible documentation: Using Ansible collections (opens in a new tab).
• Try: put bookshop.yml's tasks into roles, the way Lesson 7 did for site.yml: users, ssh, firewall, database, app, web, backup. Does the audit still pass, and is the playbook easier to read?
• Try: break something the audit checks, like chmod 755 /home/sam on bookshop3. Which runs first, the audit's FAIL or the playbook's fix?