Loops, conditions and one-liners

The problem: the same thing, many times

So far you've typed one command at a time. Real work is rarely like that: copy every .csv file, check every service, read every user, wait until the app is back after a restart, run the same check on ten servers. Typing each one by hand is slow and easy to get wrong. The shell has loops and conditions built in, and a few lines of them, often on one line, do the lot. That's a one-liner. Lesson 13 puts the same pieces into scripts.

Everything here runs as Sam on the server, except the first and last parts. On your own computer:

ssh bookshop

You land as sam@bookshop. Lesson 11 set this up, so no Vagrant is needed. Make a folder to play in:

mkdir -p ~/loops && cd ~/loops

Every command says whether it worked

When a command finishes, it leaves a number behind: its exit status. 0 means it worked. Anything else means it didn't. The shell keeps the last one in $?:

true; echo "true gave $?"
false; echo "false gave $?"
grep -q nginx /etc/passwd; echo "grep gave $?"
ls /nope; echo "ls gave $?"

true gave 0
false gave 1
grep gave 1
ls: cannot access '/nope': No such file or directory
ls gave 2

• true and false do nothing except succeed and fail. They're useful for testing.
• grep -q prints nothing (-q is quiet) and only answers through its exit status: 0 if it found the text, 1 if it didn't. It gave 1: there's no user called nginx, because on Ubuntu nginx runs as www-data (Lesson 5). A quiet grep is how you ask a yes-or-no question.
• ls gave 2, not 1. Programs use different non-zero numbers for different failures, and man ls lists them. For deciding what to do next, all that matters is zero or not zero.

$? changes after every command, including echo, so read it straight away.

&& and ||: do this only if

Lessons 3 and 4 started with cd linux-practice && docker run .... Now you know what && was doing: run the second command only if the first one's exit status was 0. || is the opposite: run it only if the first one failed. Together they make a quick either-or:

systemctl is-active nginx books-api postgresql
[ -d /srv/books-data ] && echo "the data folder exists" || echo "no data folder"
[ -f /nope.conf ] && echo "found it" || echo "no such file"

active
active
active
the data folder exists
no such file

systemctl is-active prints one line per service, in the order you named them: all three of the bookshop's are running.

[ ] is a test. It prints nothing, and only sets an exit status. The spaces inside the brackets are required: [-d /srv] is an error. The tests you'll use most:

Numbers use -ge and friends, not >: inside [ ], > would redirect output to a file (Lesson 4).

a && b || c has a catch. c runs if a fails, but also if b fails. With echo as b, that never happens, so the pattern's fine for messages. When the middle part does real work, use if, which keeps the two sides apart:

if [ -d /srv/books-data ]; then
    echo "the data folder exists"
else
    echo "no data folder"
fi

if runs any command and checks its exit status: if grep -q ..., if systemctl is-active --quiet nginx. [ ] is just the command you'll use with it most. The block ends with fi, which is if backwards. You can type it on one line, with semicolons: if [ -d /srv/books-data ]; then echo yes; else echo no; fi.

for: once for each thing

Make some files to work on. One has a space in its name, on purpose:

touch "sales report.csv" orders.csv returns.csv
for f in *; do echo "file: $f"; done

file: orders.csv
file: returns.csv
file: sales report.csv

A for loop runs the commands between do and done once for each item, with the variable (f here, any name works) set to that item. * gives every file name in the folder (Lesson 3), and $f is how you use the variable. Lists work too: for s in nginx postgresql, or {1..5} for the numbers 1 to 5:

for i in {1..5}; do curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1/; done | sort | uniq -c

      5 200

Five requests to the bookshop site, and five 200s (Lesson 9). The whole loop's output goes into the pipe, like a single command's would. sort | uniq -c (Lesson 4) counts them: with a hundred requests, you'd see at once if any had failed.

The space trap

Every $f above is in double quotes. Here's why. A common way to loop over files is to use the output of ls, with $( ), which runs a command and puts its output in its place:

for f in $(ls); do echo "file: $f"; done

file: orders.csv
file: returns.csv
file: sales
file: report.csv

sales report.csv became two files, sales and report.csv, and neither of them exists. $(ls) hands the loop one long piece of text, and the shell splits it at every space. The same happens to an unquoted variable: $f with a space in it becomes two words, so wc -l $f is given two names. Double quotes keep a value in one piece. Two rules cover it:

• Loop over *, never over $(ls). The shell's own * gives each name whole, spaces and all.
• Always write "$f", in double quotes. Single quotes would stop the variable working at all: '$f' is just the two characters $f.

Try every combination:

pass 1: wc -l [orders.csv]
pass 2: wc -l [returns.csv]
pass 3: wc -l [sales] (no such file)
pass 4: wc -l [report.csv] (no such file)

Echoing names is harmless. Deleting isn't, as "Spot the problem" shows.

Done safely, a loop can change each name. Make a backup copy of every .csv, with .bak in place of .csv:

for f in *.csv; do cp "$f" "${f%.csv}.bak"; done; ls

 orders.bak   orders.csv   returns.bak   returns.csv  'sales report.bak'  'sales report.csv'

${f%.csv} is $f with .csv cut off the end, so orders.csv becomes orders, and .bak is added on. ls shows 'sales report.bak' in quotes so you can see where the name starts and ends.

while read: once for each line

for is for lists of names. To work through a file line by line, use while read. List the real people's accounts on this server: users with a user ID (uid) of 1000 or more. /etc/passwd (Lesson 5) has one user per line, with fields separated by ::

while IFS=: read -r user _ uid _ _ home shell; do [ "$uid" -ge 1000 ] && echo "$user $home $shell"; done < /etc/passwd

nobody /nonexistent /usr/sbin/nologin
vagrant /home/vagrant /bin/bash
sam /home/sam /bin/bash

• < /etc/passwd after done feeds the file into the whole loop.
• read takes one line each time round, and the loop stops when there are no lines left.
• IFS=: says the fields are separated by :, just for this read. read puts each field into the next name: the 1st into user, the 3rd into uid, the 6th into home, and the rest of the line into the last one, shell. _ is the usual name for "a field I don't need".
• -r stops read treating \ as special. Always use it.

The test kept only lines with a uid of 1000 or more. vagrant and sam are the two real accounts. nobody is a special account with uid 65534, which owns nothing and can't log in: its shell is /usr/sbin/nologin. Checking your results catches surprises like that.

until: wait for something

Restart the app, and wait until it's answering before doing anything else. until is a loop that keeps going until a command succeeds. sudo asks for Sam's password, so run this by itself first, and type it:

sudo -v

-v asks for the password now, and sudo remembers it for a few minutes. Then:

sudo systemctl restart books-api; until curl -s -o /dev/null http://127.0.0.1:8080/; do echo "waiting..."; sleep 0.2; done; echo "books-api is up"

waiting...
books-api is up

curl fails, with a non-zero exit status, while nothing is listening on port 8080. The loop printed waiting... once, slept a fifth of a second, tried again, and curl worked: the app takes well under a second to start. while is the same loop the other way round: it keeps going while a command succeeds.

Always put a sleep in a waiting loop. Without one, it tries again thousands of times a second and keeps a whole CPU busy. And on a real server, put a limit on it, so a service that never comes back doesn't leave you waiting forever: for i in {1..30}; do curl -s ... && break; sleep 1; done. break leaves a loop early.

find and xargs: many files, wherever they are

* only sees one folder. To work on files anywhere inside a folder, use find (Lesson 3), and hand its results to a command with xargs, which runs the command with them:

find ~/loops -name "*.csv" -print0 | xargs -0 wc -l

0 /home/sam/loops/sales report.csv
0 /home/sam/loops/orders.csv
0 /home/sam/loops/returns.csv
0 total

All three, with sales report.csv in one piece, and every count is 0 because touch made them empty. -print0 and -0 separate the names with a special invisible character instead of new lines or spaces, so the space trap can't happen. Use them as a pair, every time. xargs puts many names on one command, which is why wc printed a total.

find ... -exec wc -l {} + does the same without xargs: {} stands for the names, and + means "as many at a time as fit".

Your own shortcuts: aliases and functions

An alias is a short name for a longer command: lp for cd ~/Apps/linux-practice, say, so one word takes you to your practice folder. A function is the same idea, with room for arguments and more than one command. Each user's shortcuts live in a file their shell reads when it starts: ~/.bashrc for bash on the server, ~/.zshrc for zsh on a Mac. Give Sam two:

echo 'alias ll="ls -lh"' >> ~/.bashrc
echo 'up() { systemctl is-active "$@"; }' >> ~/.bashrc
source ~/.bashrc
type ll up
up nginx books-api
exit

ll is aliased to `ls -lh'
up is a function
up () 
{ 
    systemctl is-active "$@"
}
active
active
logout
Connection to 192.168.34.130 closed.

• >> adds to the end of the file. A single > would replace everything already in it.
• source reads the file again now. Otherwise, the new shortcuts only work from the next login.
• type tells you what a name really is: an alias, a function or a program. It's the first thing to run when a command does something you didn't expect.
• "$@" inside a function means "every argument it was given, each kept whole", so up nginx books-api runs systemctl is-active nginx books-api. The quotes matter for the same reason as "$f".

The exit took you back to your own computer. Its shell has its own file, and its own shortcuts. With an lp alias in ~/.zshrc:

type lp

lp is an alias for cd ~/Apps/linux-practice

A loop across servers

A loop doesn't care what the command is, and that includes ssh. On your own computer:

for h in bookshop; do ssh "$h" 'echo "$(hostname): $(uptime -p)"'; done

bookshop: up 3 hours, 40 minutes

There's only one server in the list here; with ten names, the same line checks all ten. Two things make this work:

• The single quotes around the command stop your own computer's shell touching it. The whole thing travels to the server as typed, and $(hostname) runs there. In double quotes, your computer would fill in $(hostname) first, and you'd get your computer's name from every server.
• uptime -p shows how long the server's been running, in words.

Spot the problem

Work out what's wrong before you read the explanation.

The clean-up that deleted the wrong file. To clear out old logs, someone runs for f in $(ls *.log); do rm $f; done in a folder holding old app.log and the live app.log. Afterwards, app.log is gone too, and rm complained about a file called old. Why? $(ls) split old app.log into old and app.log, and the unquoted $f would have split it anyway. So rm was told to delete old, which didn't exist, and app.log, which did. Both rules again: for f in *.log; do rm "$f"; done, and a dry run with echo first.

Where you are now

You can make the shell decide and repeat: act on success or failure, test files and numbers, loop over files, lines, numbers and servers, and wait for a service, all without the space trap. Typing these each time gets old, and some jobs need to run at 3am when you're asleep. Lesson 13 puts them into shell scripts that stop safely when something fails, and runs them on a schedule with cron.

Check yourself

Key terms

• Exit status: the number a command leaves when it finishes. 0 means it worked.
• Test: [ ], a command that only sets an exit status, for files, text and numbers.
• Loop: commands that run again and again: for each item, while or until a command succeeds.
• Command substitution: $( ), a command whose output is put in its place.
• Word splitting: the shell breaking unquoted text at spaces into separate words. Double quotes stop it.
• Alias and function: your own short names for commands. A function can take arguments.
• One-liner: a loop or chain of commands typed on a single line.

Go deeper

• help for, help while and help test in bash: the built-in pages for these, shorter than man bash.
• Add an alias of your own to ~/.zshrc on your computer, source ~/.zshrc, and check it with type.
• Count how many lines each file in /etc/ssh has, in one line: for f in /etc/ssh/*; do [ -f "$f" ] && echo "$(wc -l < "$f") $f"; done.
• Lesson 4, Working with text, for the pipes these loops feed into.