Shell scripts and cron, done properly

The problem: the job nobody watches

The bookshop's database has no backup. A backup is the classic job for a script: the same steps every night, at a time nobody's awake. That's exactly what makes it dangerous. A scheduled job is an alarm clock going off in an empty room: if it fails and nothing is listening, nobody finds out until the day you need it.

That's not a made-up worry. In January 2017, GitLab.com lost about six hours of its production database. Their nightly pg_dump backups had been failing for a while, because the backup job ran an older pg_dump than the database needed. The job did send emails about it, but the emails were being rejected, so, in their words, "we were never aware of the backups failing, until it was too late." Every part of this lesson is about not being that team: scripts that stop when something goes wrong, output that goes somewhere you'll see it, and a backup you've actually restored.

On your own computer:

ssh bookshop

A first script

A script is a file of commands, run from top to bottom, like typing them yourself. Make a folder for them, and a first one. cat > file <<'EOF' writes everything up to the EOF line into the file (a heredoc); the quotes around 'EOF' stop the shell filling in $ values while it writes:

mkdir -p ~/scripts && cd ~/scripts
cat > hello.sh <<'EOF'
#!/bin/bash
echo "I am $0, given $# arguments"
for a in "$@"; do echo "  argument: $a"; done
EOF
./hello.sh one two

-bash: ./hello.sh: Permission denied

A new file isn't executable until you say so (Lesson 5). ./ means "the one in this folder": the shell doesn't look in the current folder for programs, on purpose. Add the x:

chmod +x hello.sh && ls -l hello.sh
./hello.sh one "two three"

-rwxrwxr-x 1 sam sam 93 Oct 30 16:35 hello.sh
I am ./hello.sh, given 2 arguments
  argument: one
  argument: two three

• #!/bin/bash, the first line, is the shebang. It tells the kernel which program runs this file. Without it, you'd get whichever shell happened to run it, and that matters later in this lesson.
• $0 is the script's own name, $# how many arguments it was given, and $1, $2 each one. "$@" is all of them, each kept whole, as in Lesson 12's function: "two three" arrived as one argument.

Make it stop when something fails

Here's a script that does what most scripts do: change folder, then act there. It only prints what it would delete, so it's safe to run:

cat > unsafe.sh <<'EOF'
#!/bin/bash
cd /nope
echo "now deleting everything in $(pwd)"
echo "backing up to $BACKUP_DIR/"
false | true
echo "the pipe gave $?"
EOF
chmod +x unsafe.sh && ./unsafe.sh; echo "script exit: $?"

./unsafe.sh: line 2: cd: /nope: No such file or directory
now deleting everything in /home/sam/scripts
backing up to /
the pipe gave 0
script exit: 0

Read that slowly. The cd failed, and the script carried on anyway, in the wrong folder: if that echo had been rm -rf *, it would have emptied Sam's scripts folder. $BACKUP_DIR was never set, so it quietly became nothing, and the backup "went to" /. A pipe whose first command failed reported success. And the script as a whole said 0, it worked. Every one of those is a real way scripts have deleted the wrong things. By default, bash keeps going.

Three settings change that. Add them as the second line:

sed 's|^#!/bin/bash|#!/bin/bash\nset -euo pipefail|' unsafe.sh > safe.sh && chmod +x safe.sh && head -3 safe.sh
./safe.sh; echo "script exit: $?"

#!/bin/bash
set -euo pipefail
cd /nope
./safe.sh: line 3: cd: /nope: No such file or directory
script exit: 1

Stopped at the first failure, and reported failure. Point the cd somewhere real, and the next problem stops it too:

sed -i 's|^cd /nope|cd /tmp|' safe.sh && ./safe.sh; echo "script exit: $?"

now deleting everything in /tmp
./safe.sh: line 5: BACKUP_DIR: unbound variable
script exit: 1

Start every script with set -euo pipefail. -e has exceptions: it doesn't stop for a command in an if, or on the left of && or ||, because there you're already checking it yourself. That's what you want.

Clean up, even when it fails

Scripts often make temporary files. If the script stops halfway, they're left behind. trap, which you met with signals in Lesson 6, runs a command when the script exits, however it exits:

cat > cleanup.sh <<'EOF'
#!/bin/bash
set -euo pipefail
tmp=$(mktemp)
trap 'rm -f "$tmp"; echo "cleaned up $tmp"' EXIT
echo "working in $tmp"
false
echo "never printed"
EOF
chmod +x cleanup.sh && ./cleanup.sh; echo "script exit: $?"

working in /tmp/tmp.v6J978zZHW
cleaned up /tmp/tmp.v6J978zZHW
script exit: 1

mktemp made a file with a random name, so two copies of the script can't collide. false failed, -e stopped the script, and the EXIT trap still ran and removed the file. The script still reported failure: the clean-up doesn't hide it.

The real job: back up the bookshop

Before writing a backup, check what there is to back up. As in Lesson 8, sudo -u postgres runs a command as the database's own admin user. If sudo hasn't asked for Sam's password in a while, run sudo -v on its own first, so nothing you paste after it gets typed into the prompt:

sudo -u postgres psql -d books -c '\dt'

Did not find any relations.

The books database is empty: no tables. Give it one, with three books:

sudo -u postgres psql -d books -c "CREATE TABLE books (id serial PRIMARY KEY, title text NOT NULL, stock int NOT NULL DEFAULT 0);" -c "INSERT INTO books (title, stock) VALUES ('The Linux Command Line', 4), ('Designing Data-Intensive Applications', 2), ('The Phoenix Project', 7);" -c "SELECT * FROM books;"

CREATE TABLE
INSERT 0 3
 id |                 title                 | stock 
----+---------------------------------------+-------
  1 | The Linux Command Line                |     4
  2 | Designing Data-Intensive Applications |     2
  3 | The Phoenix Project                   |     7
(3 rows)

pg_dump writes a database out as SQL: the commands to build it again. A peek at the start of one:

sudo -u postgres pg_dump books | head -12

--
-- PostgreSQL database dump
--

\restrict nqmazWJgdX8V2rhe9LDVsgKqrJkE3eM5QufN8haDobgIhZ8SWhGZRnNPPD0Qfz9

-- Dumped from database version 16.15 (Ubuntu 16.15-0ubuntu0.24.04.1)
-- Dumped by pg_dump version 16.15 (Ubuntu 16.15-0ubuntu0.24.04.1)

SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;

The \restrict line is a recent safety feature: while the file is being restored, it stops anything hidden in the data from running commands on your computer. Note the two version lines, too: GitLab's backups failed because those didn't match.

Now the script. It goes in /usr/local/bin, the place for programs you add yourself, which is already in everyone's PATH (Lesson 1), so it runs by name. Root has to write it there, so sudo tee (Lesson 7):

sudo tee /usr/local/bin/books-backup > /dev/null <<'EOF'
#!/bin/bash
# Back up the books database to the data disk. Keeps 7 days.
set -euo pipefail
umask 077

dest=/srv/books-data/backups
file="$dest/books-$(date +%Y%m%d-%H%M%S).sql.gz"
tmp="$file.partial"

[ "$(id -u)" -eq 0 ] || { echo "books-backup: run me with sudo" >&2; exit 1; }
trap 'rm -f "$tmp"' EXIT

install -d -m 700 "$dest"
runuser -u postgres -- pg_dump books | gzip > "$tmp"
mv "$tmp" "$file"
find "$dest" -name 'books-*.sql.gz' -mtime +7 -delete
echo "$(date '+%F %T') backup written: $file ($(du -h "$file" | cut -f1))"
EOF
sudo chmod 755 /usr/local/bin/books-backup && ls -l /usr/local/bin/books-backup

-rwxr-xr-x 1 root root 520 Oct 30 16:36 /usr/local/bin/books-backup

Line by line:

• umask 077 makes every file the script creates private to root. A backup holds all your data, so it gets the same care as the database.
• The file name has the date and time in it, so each backup has its own name and they sort in order.
• The root check stops at once, with a clear message, if someone forgets sudo. >&2 sends the message to the error output, where messages about problems belong.
• It writes to .partial first, and renames when finished. mv within one disk is instant, so a file without .partial is always a whole backup. If pg_dump fails halfway, pipefail stops the script before the mv, and the trap deletes the half-written file. You'll never mistake a broken backup for a good one.
• install -d -m 700 makes the folder, readable only by root.
• runuser -u postgres -- is sudo -u postgres for a script that's already root: it doesn't ask for anything.
• find ... -mtime +7 -delete removes backups older than 7 days, so they can't fill the disk (Lesson 10).

Check it before you trust it. bash -n reads a script without running it, and reports syntax errors. shellcheck goes much further: it knows hundreds of common script mistakes. Install it, and run it on the first version of this script, which had mkdir -p -m 700 "$dest" where install -d is now:

bash -n /usr/local/bin/books-backup && echo "syntax OK"
sudo apt-get install -y -qq shellcheck > /dev/null && shellcheck /usr/local/bin/books-backup && echo "shellcheck: no problems"

syntax OK

In /usr/local/bin/books-backup line 12:
mkdir -p -m 700 "$dest"
         ^-- SC2174 (warning): When used with -p, -m only applies to the deepest directory.

For more information:
  https://www.shellcheck.net/wiki/SC2174 -- When used with -p, -m only applie...

A real catch: with -p, mkdir only sets the mode on the last folder it makes. Harmless here, wrong in general, and exactly the kind of thing nobody spots by eye. With install -d instead, shellcheck says shellcheck: no problems. Run it on every script you write.

Run it, first without sudo, then with:

books-backup; echo "exit: $?"
sudo books-backup; echo "exit: $?"
sudo sh -c 'zcat /srv/books-data/backups/books-*.sql.gz | grep -A4 "^COPY"'

books-backup: run me with sudo
exit: 1
2026-10-30 16:36:30 backup written: /srv/books-data/backups/books-20261030-163629.sql.gz (4.0K)
exit: 0
COPY public.books (id, title, stock) FROM stdin;
1	The Linux Command Line	4
2	Designing Data-Intensive Applications	2
3	The Phoenix Project	7
\.

The three books, inside the backup. zcat reads a .gz file without unpacking it to disk. sudo sh -c '...' runs the whole line as root, * included: Sam can't list the private backups folder, so Sam's own shell couldn't expand the *.

Run it every night: cron

cron has been on Unix servers since the 1970s. Every minute, it checks its schedules and runs whatever is due. System jobs go in files in /etc/cron.d/, one job per line:

30  2  *  *  *   root   /usr/local/bin/books-backup >> /var/log/books-backup.log 2>&1
│   │  │  │  │   │      └ the command
│   │  │  │  │   └ which user runs it
│   │  │  │  └ day of the week, 0 to 7 (0 and 7 are Sunday)
│   │  │  └ month, 1 to 12
│   │  └ day of the month, 1 to 31
│   └ hour, 0 to 23
└ minute, 0 to 59

That's 2:30am every day. * means "every". */15 in the minute field means every 15 minutes, 1-5 in the weekday field Monday to Friday. The job runs when all the fields match. Build some:

30 2 * * *  root  /usr/local/bin/books-backup

next runs, from Fri 30 Oct 16:50 UTC:
Sat 31 Oct  02:30
Sun 1 Nov  02:30
Mon 2 Nov  02:30
Tue 3 Nov  02:30
Wed 4 Nov  02:30

To see cron work without waiting until 2:30am, add the backup every minute for a moment, and a second line that saves the environment cron gives its jobs:

printf '%s\n' '* * * * * root env > /tmp/cron-env.txt' '* * * * * root /usr/local/bin/books-backup >> /var/log/books-backup.log 2>&1' | sudo tee /etc/cron.d/books-backup
ls -l /etc/cron.d/books-backup
date; sleep 75; date

* * * * * root env > /tmp/cron-env.txt
* * * * * root /usr/local/bin/books-backup >> /var/log/books-backup.log 2>&1
-rw-r--r-- 1 root root 116 Oct 30 16:36 /etc/cron.d/books-backup
Fri Oct 30 04:36:30 PM UTC 2026
Fri Oct 30 04:37:45 PM UTC 2026

No restart, no reload: cron notices new files by itself. The file must belong to root, and not be writable by anyone else, or cron ignores it, because anything in it runs as whoever it names. Now look:

cat /tmp/cron-env.txt
sudo cat /var/log/books-backup.log

HOME=/root
LOGNAME=root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
LANG=en_US.UTF-8
SHELL=/bin/sh
PWD=/root
2026-10-30 16:37:01 backup written: /srv/books-data/backups/books-20261030-163701.sql.gz (4.0K)

At 16:37:01, cron ran the backup. >> /var/log/books-backup.log 2>&1 is what made it visible: >> adds the output to a log file, and 2>&1 (Lesson 4) sends error messages to the same place. Without that, where would the output go? Hold that thought.

cron's environment is not yours. Six variables, against the dozens in your login. No aliases, no functions from ~/.bashrc, a different home and folder. The PATH here is a full one only because Ubuntu sets it up for cron, from /etc/environment:

cat /etc/environment
grep -n pam_env /etc/pam.d/cron

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
8:# Read environment variables from pam_env's default files, /etc/environment
9:# and /etc/security/pam_env.conf.
10:session       required   pam_env.so

On other Linux systems, cron's PATH is often just /usr/bin:/bin, and a script that calls runuser (in /usr/sbin) works by hand and fails under cron. Safe habit: in scripts meant for cron, use full paths, or set PATH at the top.

Turn the every-minute job off now. Left alone, it makes a backup every minute: while preparing this lesson, it made twelve before it was stopped. sudo rm /etc/cron.d/books-backup.

Two cron traps, on purpose

Three test lines, which only print the time or some numbers:

printf '%s\n' '* * * * * root date +%H:%M >> /tmp/cron-pct.txt 2>&1' '* * * * * root date +\%H:\%M >> /tmp/cron-pct-ok.txt 2>&1' '* * * * * root echo {1..3} >> /tmp/cron-sh.txt 2>&1' | sudo tee /etc/cron.d/cron-traps
s=$(date +%-S); echo "waiting $((62 - s)) seconds..."; sleep $((62 - s))
cat /tmp/cron-pct.txt; echo "---"; cat /tmp/cron-pct-ok.txt; echo "---"; cat /tmp/cron-sh.txt

* * * * * root date +%H:%M >> /tmp/cron-pct.txt 2>&1
* * * * * root date +\%H:\%M >> /tmp/cron-pct-ok.txt 2>&1
* * * * * root echo {1..3} >> /tmp/cron-sh.txt 2>&1
waiting 62 seconds...
cat: /tmp/cron-pct.txt: No such file or directory
---
16:49
16:50
---
{1..3}
{1..3}

The first line made no file at all. The second, the same command with each % written as \%, worked. cron's own log shows why:

sudo journalctl -u cron --no-pager --since "16:48" | grep -v pam_unix

Oct 30 16:48:01 bookshop CRON[12706]: (root) CMD (env > /tmp/cron-env.txt)
Oct 30 16:48:01 bookshop CRON[12705]: (root) CMD (/usr/local/bin/books-backup >> /var/log/books-backup.log 2>&1)
Oct 30 16:49:01 bookshop CRON[12913]: (root) CMD (date +%H:%M >> /tmp/cron-pct-ok.txt 2>&1)
Oct 30 16:49:01 bookshop CRON[12915]: (root) CMD (date +)
Oct 30 16:49:01 bookshop CRON[12912]: (CRON) info (No MTA installed, discarding output)
Oct 30 16:50:01 bookshop CRON[12925]: (root) CMD (echo {1..3} >> /tmp/cron-sh.txt 2>&1)
Oct 30 16:50:01 bookshop CRON[12924]: (CRON) info (No MTA installed, discarding output)

(sudo, because Sam isn't in the groups that may read system logs. The two lines from 16:48 are the every-minute jobs, just before they were removed.)

• % ends the command in a cron line. cron ran just date +. Everything after the first % became the command's input, so the >> /tmp/cron-pct.txt was never part of the command. Write \%, or, better, keep anything with % in a script and call the script.
• Output nobody captured is thrown away. cron tries to email a job's output to its owner. No MTA installed, discarding output: with no mail system (MTA, mail transfer agent) on the server, it's gone. With one, it lands in a mailbox nobody reads, or, as at GitLab, gets rejected. Always send a cron job's output to a log file yourself. This is the alarm clock in the empty room.
• {1..3} stayed {1..3}. SHELL=/bin/sh, from cron's environment: cron runs command lines with sh, which doesn't know bash's extras. Inside your script, the #!/bin/bash line decides, which is one more reason to put the real work in a script.

Remove the test: sudo rm /etc/cron.d/cron-traps /tmp/cron-pct-ok.txt /tmp/cron-sh.txt.

A bug while writing this lesson. The first version of that waiting line was sleep $((62 - $(date +%S))). At 9 seconds past the minute, date printed 09, and:

-bash: 62 - 09: value too great for base (error token is "09")

Shell arithmetic reads a number starting with 0 as octal (base 8), which has no digit 9. So the wait was skipped, and the clean-up line deleted the test files before cron had run. date +%-S drops the leading zero, and $((10#$s)) forces base 10. It's a classic: 08 and 09 from date break scripts twice an hour.

A systemd timer instead

On a server with systemd, there's a newer way: a timer. It's two small unit files, like the service in Lesson 7. A service, saying what to run:

sudo tee /etc/systemd/system/books-backup.service > /dev/null <<'EOF'
[Unit]
Description=Back up the books database

[Service]
Type=oneshot
ExecStart=/usr/local/bin/books-backup
EOF

Type=oneshot means "runs, finishes, done", not a program that keeps running. And a timer, saying when:

sudo tee /etc/systemd/system/books-backup.timer > /dev/null <<'EOF'
[Unit]
Description=Back up the books database every night

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target
EOF
systemd-analyze calendar "*-*-* 02:30:00"
sudo systemctl daemon-reload && sudo systemctl enable --now books-backup.timer
systemctl list-timers books-backup.timer --no-pager

Normalized form: *-*-* 02:30:00
    Next elapse: Sat 2026-10-31 02:30:00 UTC
       From now: 9h left
Created symlink /etc/systemd/system/timers.target.wants/books-backup.timer → /etc/systemd/system/books-backup.timer.
NEXT                        LEFT LAST PASSED UNIT               ACTIVATES           
Sat 2026-10-31 02:30:00 UTC   9h -         - books-backup.timer books-backup.service

1 timers listed.
Pass --all to see loaded but inactive timers, too.

OnCalendar is year-month-day hour:minute:second, with * for "every". systemd-analyze calendar checks a schedule before you trust it, and says when it next fires: no guessing. Persistent=true means that if the server was off at 2:30am, the backup runs as soon as it's back. cron would just skip that night. Test it now, without waiting:

sudo systemctl start books-backup.service
sudo journalctl -u books-backup.service --no-pager -n 4

Oct 30 16:48:09 bookshop runuser[12857]: pam_unix(runuser:session): session closed for user postgres
Oct 30 16:48:09 bookshop books-backup[12853]: 2026-10-30 16:48:09 backup written: /srv/books-data/backups/books-20261030-164809.sql.gz (4.0K)
Oct 30 16:48:09 bookshop systemd[1]: books-backup.service: Deactivated successfully.
Oct 30 16:48:09 bookshop systemd[1]: Finished books-backup.service - Back up the books database.

The output went to the journal by itself. No >> log 2>&1 to forget, and if the script fails, systemctl status books-backup.service says failed, which monitoring tools can watch for.

Both are everywhere, so you need to read both. For new jobs on a systemd server, use a timer. The bookshop keeps the timer.

The test that matters: restore it

A backup you've never restored is a hope, not a backup. GitLab had several backup methods; when they needed them, none worked as expected, and nobody was responsible for testing them. So restore the newest backup into a scratch database, and look:

latest=$(sudo ls -t /srv/books-data/backups | head -1); echo "restoring $latest"
sudo -u postgres createdb books_restore
sudo sh -c "zcat /srv/books-data/backups/$latest" | sudo -u postgres psql -q -d books_restore
sudo -u postgres psql -d books_restore -c "SELECT * FROM books;"
sudo -u postgres dropdb books_restore

restoring books-20261030-164809.sql.gz
 set_config 
------------
 
(1 row)

 setval 
--------
      3
(1 row)

 id |                 title                 | stock 
----+---------------------------------------+-------
  1 | The Linux Command Line                |     4
  2 | Designing Data-Intensive Applications |     2
  3 | The Phoenix Project                   |     7
(3 rows)

All three books, rebuilt from the file. The set_config and setval results are the dump setting things up, such as where the id numbers carry on from. dropdb removes the scratch copy. Finally, clear out the every-minute backups, keeping only the newest, and check its permissions:

sudo sh -c 'cd /srv/books-data/backups && ls -t books-*.sql.gz | tail -n +2 | xargs rm -v' | tail -3
sudo ls -l /srv/books-data/backups

removed 'books-20261030-163801.sql.gz'
removed 'books-20261030-163701.sql.gz'
removed 'books-20261030-163629.sql.gz'
total 4
-rw------- 1 root root 873 Oct 30 16:48 books-20261030-164809.sql.gz

tail -n +2 is everything from the 2nd line on: all but the newest. And the survivor is -rw-------, private to root, thanks to umask 077. The backups made before that line was added were -rw-r--r--, readable by anyone who could reach the folder.

Spot the problem

Work out what's wrong before you read the explanation.

The report that worked by hand. A script runs fine when you type it. In cron, nothing happens, and the log file it should write to stays empty. The script's first lines are cd ~/reports and ./make-report, and it has no set -e. What do you check? Whose ~ it is. In /etc/cron.d, the job runs as the user named in the line, often root, so ~ is /root, not your home. The cd failed, the script carried on in the wrong folder, ./make-report wasn't there either, and without its output redirected, cron threw both errors away. Full paths, set -euo pipefail, and >> log 2>&1 would have made it fail loudly at line one.

Where you are now

The bookshop backs itself up every night, privately, to its own disk, keeps a week, logs every run, and you've proved a backup restores. You can write scripts that stop at the first failure, clean up after themselves, and run unattended, by cron or by a systemd timer. Lesson 14 puts all twelve lessons before it to work: a method for troubleshooting, and six broken servers to fix with it.

Check yourself

Key terms

• Script: a file of commands, run top to bottom.
• Shebang: the #! first line, naming the program that runs the script.
• set -euo pipefail: stop on any failure, on unset variables, and on any failed part of a pipe.
• trap: a command to run when the script exits, or gets a signal.
• umask: the setting that decides the permissions new files start with.
• cron: the classic scheduler, reading job lines from /etc/cron.d and crontabs.
• systemd timer: a unit that starts a service on a schedule.
• Heredoc: <<'EOF', several lines of text fed to a command.

Go deeper

• crontab -e edits a user's own cron schedule, without the user field. crontab -l lists it. ls /etc/cron.daily shows the jobs Ubuntu already runs every day.
• systemctl list-timers with no name shows every timer on the server, including the system's own.
• man 5 crontab is the full cron line format, including names like @daily and @reboot.
• GitLab's own write-up: Postmortem of database outage of January 31 (opens in a new tab), worth reading in full.
• Lesson 7, Services and logs, for unit files and the journal.