Secrets with Ansible Vault

A password with nowhere to go

The bookshop's web servers will run an app that talks to the database on db1. The app reads its settings from a file, /etc/bookshop/app.env, and one of those settings is the database password. Ansible should write that file, like every other file on the servers. So the password has to be somewhere Ansible can read it: in the repository.

And that's the problem. A git repository is copied, whole, to every computer that clones it, with every old version kept. A password committed once is in the history for good, even after you delete it, and on every laptop and backup that has a copy. If the repository is ever made public, or a laptop is lost, the password goes with it. This happens often enough that GitHub scans public repositories for leaked keys.

Ansible Vault is Ansible's answer. It encrypts files, or single values, with a password of your choice. Think of a locked box in a shared office. Everyone can see the box, move it and make copies of it, but only someone with the key can open it. Git holds the box; the key stays in your pocket.

The vault password

The key first. It goes in a file, readable only by you, which git is told to ignore:

printf 'bookshop-practice\n' > .vault_pass
chmod 600 .vault_pass
printf '.vault_pass\n' >> .gitignore
ls -l .vault_pass; git check-ignore -v .vault_pass

-rw-------  1 you  staff  18 Nov  2 21:32 .vault_pass
.gitignore:4:.vault_pass	.vault_pass

-rw-------: only you can read or write it, from Linux Lesson 5. git check-ignore -v asks git whether it would ignore a file, and why: line 4 of .gitignore. Check this before you put a real password in the file, not after. (The practice repository's .gitignore already has the line, so if you're following along, skip the third command.)

Where the secret goes

The web group's variables move from a file into a folder. Ansible reads every file in group_vars/web/ for the web group, so the settings can be split in two:

# group_vars/web/vars.yml
# Settings for the web servers, shared by every playbook that runs on them.
# The nginx role has its own default of 80; this is what health.yml reads.
http_port: 80

# The app's database password. The real value is in vault.yml, encrypted.
db_password: "{{ vault_db_password }}"

# group_vars/web/vault.yml, before it's encrypted
vault_db_password: Cranberry-Shelf-91

This is the usual layout, and the reason for it is searching. Once vault.yml is encrypted, nobody can search inside it, or see what's in it without the password. vars.yml stays readable, so anyone searching the repository for db_password finds it, sees where its value comes from, and knows which file to open. The vault_ prefix says "this one's in the vault".

Encrypting it

ansible-vault encrypt --vault-password-file .vault_pass group_vars/web/vault.yml
cat group_vars/web/vault.yml

$ANSIBLE_VAULT;1.1;AES256
33626236316364643232663730653765376332373065636164363461376235613735363730626131
3036646539336165656531363564663966363130333764610a386366666638396464626230333763
66323162383730623435326365363930393064656462363030326531363039656531313661323731
...
3966323236666532383464393837636664663063313165333864

That's what git will store. The first line says it's a vault file, format 1.1, encrypted with AES-256 (the Advanced Encryption Standard, with a 256-bit key: the same kind of encryption banks use). The rest is the scrambled contents. Without the vault password, it can't be read. With it:

ansible-vault view --vault-password-file .vault_pass group_vars/web/vault.yml

# Encrypted with Ansible Vault. Edit with: ansible-vault edit group_vars/web/vault.yml
vault_db_password: Cranberry-Shelf-91

The whole file was encrypted, including that comment. So the comment only helps someone who can already open the file; the note that helps everyone else is the one in vars.yml. Three commands cover nearly all the work:

• ansible-vault view shows the contents.
• ansible-vault edit opens it in your editor, decrypted, and encrypts it again when you save. The plain text is never written to the repository.
• ansible-vault create starts a new encrypted file the same way.

A role that uses it

The new app role writes the app's settings:

# roles/app/tasks/main.yml
- name: Make the app's settings folder
  ansible.builtin.file:
    path: /etc/bookshop
    state: directory
    owner: root
    group: root
    mode: "0755"

- name: Write the app's settings, readable only by root
  ansible.builtin.template:
    src: app.env.j2
    dest: /etc/bookshop/app.env
    owner: root
    group: root
    mode: "0600"

# roles/app/templates/app.env.j2
# {{ ansible_managed }}
# Settings for the bookshop app on {{ inventory_hostname }}.
DB_HOST=db1
DB_PORT={{ hostvars['db1']['db_port'] }}
DB_USER=bookshop
DB_PASSWORD={{ db_password }}

Two new things. mode: "0600" means only root can read the file. And hostvars['db1']['db_port'] reads a variable belonging to a different server: the database port from host_vars/db1.yml, from Lesson 4, used on the web servers. hostvars holds every server's variables, so one server's settings can depend on another's.

The web play gets the role, one line under nginx:

- name: Web servers
  hosts: web
  become: true
  roles:
    - nginx
    - app

Running without the key

ansible-playbook site.yml --limit web1

PLAY [Baseline for every bookshop server] **************************************
ERROR! Attempting to decrypt but no vault secrets found

It stops straight away, before any task runs on any server. Ansible loads group_vars at the start, found an encrypted file, and had no password to open it with. Nothing half-done, nothing to clean up. With the password:

ansible-playbook site.yml --limit web1 --vault-password-file .vault_pass --diff

TASK [app : Write the app's settings, readable only by root] *******************
--- before
+++ after: /Users/you/.ansible/tmp/ansible-local-45831z_91l08p/tmp2_5mbe92/app.env.j2
@@ -0,0 +1,6 @@
+# Ansible managed
+# Settings for the bookshop app on web1.
+DB_HOST=db1
+DB_PORT=5432
+DB_USER=bookshop
+DB_PASSWORD=Cranberry-Shelf-91

changed: [web1]

The file was written, with db1's port from hostvars. And there's the password, printed on the screen. --diff shows every change a task makes, and the change here is the password. In your own terminal, that might be fine. In a CI/CD (continuous integration and delivery) pipeline, whose logs everyone on the team can read and which are kept for months, it's a leak. The vault kept the password out of git, and --diff printed it anyway.

Keeping it out of the output

One line on the task, no_log: true:

- name: Write the app's settings, readable only by root
  ansible.builtin.template:
    src: app.env.j2
    dest: /etc/bookshop/app.env
    owner: root
    group: root
    mode: "0600"
  no_log: true

The same run, on web2, which doesn't have the file yet:

ansible-playbook site.yml --limit web2 --vault-password-file .vault_pass --diff

TASK [app : Write the app's settings, readable only by root] *******************
changed: [web2]

changed, and nothing else. no_log hides everything the task would print: the diff, and also its error message if it fails. That's the trade-off: put it on the tasks that handle secrets, and no others, or a failure somewhere else will tell you nothing about why.

Not typing the password

Passing --vault-password-file every time gets old. ansible.cfg can say where it is:

[defaults]
inventory = inventory.ini
# The vault password, in a file git ignores.
vault_password_file = .vault_pass

ansible-playbook site.yml
ansible-playbook health.yml

PLAY RECAP *********************************************************************
db1                        : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web1                       : ok=15   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
web2                       : ok=15   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

PLAY RECAP *********************************************************************
web1                       : ok=8    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
web2                       : ok=8    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   

ok=15 on the web servers: Lesson 7's 13, plus the app role's two tasks. Both files are in place, so changed=0, and the health check still passes.

On the server

ansible web1 -m command -a "ls -l /etc/bookshop/app.env" --become
ansible web1 -m command -a "cat /etc/bookshop/app.env" --become
ansible web1 -m command -a "cat /etc/bookshop/app.env" --become --become-user deploy

web1 | CHANGED | rc=0 >>
-rw------- 1 root root 132 Nov  3 02:32 /etc/bookshop/app.env

web1 | CHANGED | rc=0 >>
# Ansible managed
# Settings for the bookshop app on web1.
DB_HOST=db1
DB_PORT=5432
DB_USER=bookshop
DB_PASSWORD=Cranberry-Shelf-91

web1 | FAILED | rc=1 >>
cat: /etc/bookshop/app.env: Permission deniednon-zero return code

(The third command also printed a warning about a temporary folder for deploy, which is harmless.) Owned by root, -rw-------, and deploy can't read it. But root can, and so can anyone who can use sudo. That's the limit of Vault: it protects the secret in the repository and on the way to the server. Once it's there, it's protected by file permissions, and by who has access to the server. Vault doesn't change who can log in.

One value, not a whole file

Sometimes one secret in an otherwise ordinary file is neater than a separate vault file. encrypt_string encrypts a single value, ready to paste into YAML:

ansible-vault encrypt_string "not-a-real-token" --name api_token

api_token: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          37386662626635643466396135316134313730313261343435666565383236626138306266306434
          3539653032306661353432653834643632393765316366660a646435306464663436303931653637
          ...

!vault | tells Ansible the value is encrypted. The rest of the file stays readable, and searchable, and only the value is hidden. The cost: you can't use ansible-vault view or edit on it, so changing it means encrypting a new value and pasting it in again. For more than one or two secrets, a vault file is easier.

Changing the vault password

If someone who knew the vault password leaves, or it's been shared too widely, change it. rekey re-encrypts with a new one:

printf 'a-new-practice-password\n' > /tmp/new_vault_pass
ansible-vault rekey --new-vault-password-file /tmp/new_vault_pass group_vars/web/vault.yml
ansible-vault view group_vars/web/vault.yml
ansible-vault view --vault-password-file /tmp/new_vault_pass group_vars/web/vault.yml

Rekey successful

ERROR! Decryption failed (no vault secrets were found that could decrypt) on group_vars/web/vault.yml for group_vars/web/vault.yml

# Encrypted with Ansible Vault. Edit with: ansible-vault edit group_vars/web/vault.yml
vault_db_password: Cranberry-Shelf-91

After the rekey, the old password in .vault_pass no longer opens it; the new one does. Put the practice one back, so the repository keeps working:

ansible-vault rekey --vault-password-file /tmp/new_vault_pass --new-vault-password-file .vault_pass group_vars/web/vault.yml
rm /tmp/new_vault_pass

Rekey successful

But be clear about what that does and doesn't do. Anyone who copied the repository before still has the old encrypted file, in git's history, and the old password opens it. Rekeying protects the file from now on. If the old vault password got out, change the secrets inside the vault too: here, the database password itself.

Spot the problem

Work out what's wrong before you read the explanation.

The password in the error message. A playbook sets the database user's password with a command task: ansible.builtin.command: psql -c "ALTER USER bookshop PASSWORD '{{ db_password }}'". The password comes from the vault, and .vault_pass was never committed. One night the task fails, because the database is still starting up, and the pipeline log shows the failure, including "cmd": the whole command, password and all. You've seen this happen: every failed command task in these lessons printed its cmd. The vault did its job, and the password still ended up in a log that 40 people can read. What should have been done? no_log: true on any task that uses a secret, from the start. Better still, use a module made for the job, like community.postgresql.postgresql_user, which knows its password is secret and never prints it. (A password on a command line can also be seen by other users of the server, in its list of running processes.) And since it's leaked now, change the database password.

Where you are now

The bookshop's database password is in the repository, encrypted, and on the web servers in a file only root can read. You know where Vault stops: it protects the repository, while no_log protects the output and file permissions protect the server. Lesson 9 turns to the job these lessons have been building towards: patching every server, a few at a time, with the health check deciding whether to carry on.

Check yourself

Key terms

• Secret: anything that grants access, like a password, a key or a token.
• Ansible Vault: Ansible's built-in way to encrypt files and values, opened with a vault password.
• Vault password: the password that encrypts and decrypts vault files. Kept out of git.
• AES-256: the Advanced Encryption Standard with a 256-bit key; what Ansible Vault uses.
• encrypt_string: encrypts one value, for pasting into an otherwise readable YAML file.
• Rekey: re-encrypting a vault file with a new vault password.
• no_log: a task setting that stops Ansible printing anything the task does, errors included.
• hostvars: every server's variables, so one server's tasks can read another's.

Go deeper

• The Ansible documentation: Protecting sensitive data with Ansible Vault (opens in a new tab), and Keep vaulted variables safely visible (opens in a new tab).
• Try: ansible-vault edit group_vars/web/vault.yml, change the password, and run site.yml. Which task changes, and does no_log hide what it changed?
• Try: replace db_password in vars.yml with an encrypt_string value, delete vault.yml, and run site.yml. Does anything on the servers change?