Working with text: awk, sed and pipes

The problem: lines aren't answers

In Lesson 3, grep found the lines that mattered: 37 errors, 28 server errors, 180 failed logins. But the questions people actually ask are different. Which hour was worst? Who's sending the most requests? Which pages are people looking for that don't exist? Were the slow queries the cause?

Each of those is a count, a total or a top ten, and no single command answers them. Linux's answer is a set of small tools that each do one thing, and a way to join them together.

Start the same container as Lesson 3. On your own computer, from the folder you downloaded the practice files into at the end of Lesson 2:

cd linux-practice
docker run -it --rm --name lesson4 -v "$PWD":/practice -w /practice ubuntu:24.04 bash

-v "$PWD" links whatever folder you're in, so the cd matters. Start it anywhere else, and the first command below says No such file or directory, because there's no logs folder there. Run ls inside the container to check: you should see README.md, config, data, logs and notes. And if your prompt already starts with root@ before you type the cd, you're still inside an old container: type exit first.

Everything else in this lesson runs inside the container. Anything this lesson writes goes in /tmp, so the practice files stay as they are.

Pipes: one command's output is the next one's input

You've already used one. The | (pipe) sends everything a command prints into the next command, instead of to the screen:

grep ERROR logs/app.log | head -2
grep ERROR logs/app.log | wc -l

2026-10-28 14:00:37 ERROR database connection refused: db.internal:5432
2026-10-28 14:01:03 ERROR database connection refused: db.internal:5432
37

grep found the error lines; head kept two of them, and wc -l counted them. Neither head nor wc knows anything about logs. They just work on whatever text arrives. That's the whole idea: each tool does one small job on text, and pipes chain them into something that answers a question. Most of this lesson is building pipelines one step at a time.

Columns: awk and cut

Every line of the access log has the same layout, separated by spaces. awk numbers each piece, called a field: $1 is the first, $9 the ninth, and $NF the last.

head -2 logs/access.log | awk '{print $1}'
head -2 logs/access.log | awk '{print $9, $7}'

203.0.113.24
203.0.113.23
200 /
200 /static/app.js

Read awk '{print $1}' as "for every line, print field 1". The single quotes keep the shell from touching the $. A comma between fields prints a space between them.

cut does a simpler version of the same job: -d says what separates the pieces, and -f which piece you want. Its real strength is any separator you like. Times in the access log look like [28/Oct/2026:14:00:06, so with -d: the hour is field 2. You'll use that in a moment. The catch with cut is in "Break it on purpose", below.

Counting: sort, uniq -c, sort -rn

This is the pipeline you'll type more than any other. It answers "how many of each?":

awk '{print $9}' logs/access.log | sort | uniq -c | sort -rn

   2035 200
    180 401
     57 404
     14 500
     10 503
      4 502

Step by step:

1. awk '{print $9}' prints just the status code from each of the 2,300 lines.
2. sort puts them in order, so all the 200s are together, then all the 401s, and so on.
3. uniq -c squeezes each run of identical lines into one, with a count in front.
4. sort -rn sorts again, this time by that count: -n compares numbers rather than text, and -r reverses, so the biggest comes first.

The whole day of the web server, in one view. Add | head -3 for a top three. Try it yourself, and switch stages off to see what each one does:

awk '{print $9}' access.log | sort | uniq -c

      9 200
      4 401
      2 404
      2 500
      1 502

Now swap the column, and the same pipeline answers different questions:

awk '{print $1}' logs/access.log | sort | uniq -c | sort -rn | head -3
awk '$9 == 404 {print $7}' logs/access.log | sort | uniq -c | sort -rn

    180 198.51.100.77
    120 203.0.113.200
     39 203.0.113.57
     16 /wp-login.php
     14 /robots.txt
     14 /api/book/42
     13 /old-page

The top address is the one from Lesson 3, sending 180 requests, and the second, with 120, is the ExampleBot crawler. The second command uses something new: $9 == 404 before the braces is a condition, so awk only prints the page for lines whose status is 404. And the answer is interesting. /wp-login.php is the login page of WordPress, which this shop doesn't use: that's automated scanners trying their luck, which every public server sees. /api/book/42 (singular) next to the real /api/books/42 looks like a typo somewhere, maybe in another team's code. /robots.txt is a file crawlers ask for, and adding one would stop those 404s.

Filtering and adding up with awk

A condition can be any comparison. awk treats numbers as numbers, so $9 >= 500 means status 500 and above, which is cleaner than Lesson 3's pattern:

awk '$9 >= 500 {print $4, $7, $9}' logs/access.log | head -3
awk '$9 >= 500' logs/access.log | cut -d: -f2 | sort | uniq -c
awk '$1 == "198.51.100.77" {print $4}' logs/access.log | sed -n '1p;$p'

[28/Oct/2026:14:00:06 /api/books 502
[28/Oct/2026:14:01:23 /api/books 500
[28/Oct/2026:14:02:51 /api/books 500
     28 14
[28/Oct/2026:02:10:10
[28/Oct/2026:02:40:00

• With a condition and no braces, awk prints the whole matching line.
• Which hour had the most server errors? The question Lesson 3 left open. cut -d: -f2 takes the hour from each time, and the count says all 28 were in the 14:00 hour. There was only ever one incident.
• When was the login hammering? sed -n '1p;$p' prints the first and last line (more on sed below): from 02:10:10 to 02:40:00, exactly half an hour. Text in a condition needs double quotes: $1 == "198.51.100.77".

awk can also add up. END is a block that runs once, after the last line:

awk '{sum += $10} END {print sum}' logs/access.log
awk '{sum += $10} END {print sum / 1024 / 1024, "MB"}' logs/access.log

9038881
8.62015 MB

sum += $10 adds each line's size to a running total called sum. Every variable in awk starts at zero, so there's nothing to set up. The web server sent about 8.6 megabytes in total all day.

One more trick: $NF is the last field, whatever the line's length. That suits the app log, where the number you want comes last:

grep "handled in" logs/app.log | awk '{print $NF}' | head -3

100ms
49ms
112ms

Were the slow queries the cause?

In Lesson 3, grep -v INFO showed slow database queries hours before the outage, and the question was whether they were connected. If they caused it, they should pile up near 14:00. Count them by hour:

grep "slow query" logs/app.log | cut -c12-13 | sort | uniq -c | sort -rn | head -5

      6 14
      6 12
      6 05
      5 19
      5 18

cut -c12-13 takes characters 12 and 13 of each line, which is where the hour sits in 2026-10-28 14:00:37. And the answer is no: 14:00 ties with 12:00 and 05:00, and the rest of the day has nearly as many. Slow queries are a steady background problem worth fixing, but they didn't cause the outage. This is how troubleshooting rules things out: turn a hunch into a count.

Other separators: awk -F

Settings files often use key = value. -F tells awk what separates the fields, and FILENAME is the file the line came from. One command shows every file that sets the port, and to what:

awk -F' = ' '$1 == "port" {print FILENAME, $2}' config/app/app.conf config/app/override.conf

config/app/app.conf 8080
config/app/override.conf 9090

The same trap from Lesson 3, now in a form you could put in a script.

Changing text with sed

sed (stream editor) reads text line by line and changes it on the way through. Three jobs cover most of what you'll do with it.

Print certain lines. -n means "print nothing unless told", and p means print:

sed -n '416,418p' logs/app.log
sed '/^#/d' config/app/override.conf

2026-10-28 14:17:43 INFO  user session ended
2026-10-28 14:19:38 INFO  user session started
2026-10-28 14:21:03 INFO  database connection restored
log_level = debug
port = 9090

416,418p prints lines 416 to 418: the moment the database came back, from line 418 in Lesson 3, with the two lines before it. Delete lines with d: /^#/d deletes every line that starts with #, so a settings file shows only its settings. ^ means "the start of the line".

Replace text with s/old/new/. Try it on a copy, never the real file:

cp config/app/override.conf /tmp/override.conf
sed 's/debug/info/' /tmp/override.conf
grep log_level /tmp/override.conf

# Loaded after app.conf. Anything here wins.
log_level = info
port = 9090
log_level = debug

sed printed the changed version, but the file itself still says debug. By default, sed only changes what it prints. That's a safety feature: you see the result before anything is saved. To change the file itself, add -i (in place):

sed -i 's/^log_level = debug/log_level = info/' /tmp/override.conf
grep log_level /tmp/override.conf

log_level = info

This is the fix for Lesson 3's huge debug.log: switch the log level back to info. Notice the pattern is more exact this time, ^log_level = debug, starting at the beginning of the line. s/debug/info/ would also change the word debug anywhere else it appeared, such as in a comment. sed also only replaces the first match on each line; add g at the end, s/old/new/g, to replace them all.

sed is also handy inside a pipeline, to make lines comparable before you count them. Each slow-query line ends with a different time, so they'd never count as the same:

grep WARN logs/app.log | sed 's/took [0-9]*ms/took Nms/' | cut -c21- | sort | uniq -c

     74 WARN  retrying cache connection
     79 WARN  slow query: books lookup took Nms

[0-9]* means "any number of digits", so every time becomes Nms. cut -c21- drops the date and time (everything before character 21). Now the 153 warnings are just two kinds.

Sending output somewhere: >, >> and tee

Every command has three streams: one it reads from (standard input), one for its normal output (standard output), and a separate one for error messages (standard error). Both outputs go to your screen by default, which is why they look like one thing. They're not, and that's useful.

> saves output to a file, replacing what was there. >> adds to the end:

awk '{print $9}' logs/access.log | sort | uniq -c | sort -rn > /tmp/status.txt
cat /tmp/status.txt
echo "Checked the access log" >> /tmp/status.txt
tail -2 /tmp/status.txt

   2035 200
    180 401
     57 404
     14 500
     10 503
      4 502
      4 502
Checked the access log

The first command printed nothing: its output went into the file. > creates the file if it doesn't exist, and empties it first if it does, with no warning. When in doubt, use >>.

Errors are separate. Ask ls for one file that exists and one that doesn't:

ls notes/todo.txt nothing.txt
ls notes/todo.txt nothing.txt 2> /tmp/errors.txt
cat /tmp/errors.txt

ls: cannot access 'nothing.txt': No such file or directory
notes/todo.txt
notes/todo.txt
ls: cannot access 'nothing.txt': No such file or directory

The first time, both outputs hit the screen. The second time, 2> sent standard error to a file, so only the normal output showed. The 2 is standard error's number: standard input is 0, standard output is 1, standard error is 2. A plain > is short for 1>.

Both into one file, or errors thrown away:

ls notes/todo.txt nothing.txt > /tmp/both.txt 2>&1
cat /tmp/both.txt
ls notes/todo.txt nothing.txt 2> /dev/null

ls: cannot access 'nothing.txt': No such file or directory
notes/todo.txt
notes/todo.txt

2>&1 means "send standard error wherever standard output is going", here into both.txt. /dev/null is a special file that throws away anything written to it. 2>/dev/null is what hides the "Permission denied" lines when you find across a whole server, as Lesson 3's callout mentioned.

tee saves and shows at the same time. Named after a T-shaped pipe fitting, it copies its input to a file and passes it on:

grep ERROR logs/app.log | tee /tmp/errors-only.txt | wc -l
wc -l /tmp/errors-only.txt

37
37 /tmp/errors-only.txt

The 37 error lines went into the file, and on to wc. Handy when you want a record of something and want to keep working with it.

Break it on purpose

Four mistakes everyone makes once. Run each one:

grep INFO logs/app.log | cut -d' ' -f4 | sort | uniq -c | sort -rn | head -3
grep INFO logs/app.log | awk '{print $4}' | sort | uniq -c | sort -rn | head -3

    448 
    229 user
    118 cache
    100 request

cut gave one line: 448 blanks. The app log puts two spaces after INFO, to line up with ERROR. To cut, every single space is a separator, so the second space starts an empty field, and field 4 is empty on every one of the 448 INFO lines. awk treats any run of spaces as one separator, so it found the real words: user, cache, request. For text lined up with spaces, use awk. For a single, exact separator like : or ,, cut is fine.

awk '{print $9}' logs/access.log | uniq -c | head -4

     58 200
      1 404
      2 200
      1 404

uniq without sort. uniq only joins identical lines that are next to each other. The log is in time order, not status order, so 200 turns up again and again. Always sort before uniq.

cp logs/app.log /tmp/copy.log
sort /tmp/copy.log > /tmp/copy.log
wc -l /tmp/copy.log

0 /tmp/copy.log

The file is now empty. The shell sets up > /tmp/copy.log before sort starts, and setting it up empties the file. By the time sort opens it to read, there's nothing left. Never redirect into the file you're reading. Write to a new file, then mv it over the old one, or use sort -o /tmp/copy.log /tmp/copy.log, which sort handles safely.

ls notes/todo.txt nothing.txt 2>&1 > /tmp/wrong.txt
cat /tmp/wrong.txt

ls: cannot access 'nothing.txt': No such file or directory
notes/todo.txt

The error still hit the screen. Redirections are read left to right. 2>&1 came first, when standard output was still the screen, so standard error was pointed at the screen. Only then did > move standard output to the file. > file 2>&1, in that order, is the one that sends both to the file.

Spot the problem

Work out what's wrong before you read the explanation.

The report that vanished. A nightly job runs grep ERROR app.log | sort | uniq -c > errors.txt, then emails errors.txt. One night someone changes it to sort errors.txt | uniq -c > errors.txt to tidy it, and every email after that is empty. Why? > errors.txt empties the file before sort reads it, so there's nothing to sort. Write to a new file, or use sort -o.

Where you are now

You turned the evidence into answers: one incident at 14:00, a half-hour login attack from one address, scanners and a likely typo behind the 404s, and slow queries ruled out as the cause. Lesson 5 is about who can read and change all this: users, groups and permissions, starting with the bookshop user the server should really be running as.

Check yourself

Key terms

• Pipe (|): sends one command's output into the next command's input.
• Field: one piece of a line, as awk or cut splits it. $1 is the first, $NF the last.
• Condition: in awk, a test before the braces, like $9 >= 500, that picks which lines to act on.
• Standard input, output and error: the three streams every command has: what it reads (0), its normal output (1), and its error messages (2).
• Redirection: >, >>, 2> and <, which point a stream at a file instead of the screen or keyboard.
• /dev/null: a special file that throws away anything written to it.

Go deeper

• The practice files (opens in a new tab): which page got the most 401s? Which hour of the day was busiest overall? Both are one pipeline.
• In debug.log, field 4 is the kind of event. Count them: awk '{print $4}' logs/debug.log | sort | uniq -c | sort -rn.
• The GNU awk user's guide (opens in a new tab), for everything awk can do. The container's awk is a smaller version called mawk, but everything in this lesson works in both.
• Lesson 3, Reading and searching files, for grep and find.