Reading and searching files

The problem: a bad day, and a lot of text

Something went wrong on a server yesterday, and you've been asked to find out what. All the evidence is in files: logs that record what happened, and settings that record how things were set up. Together they're tens of thousands of lines. Nobody reads that from top to bottom.

This lesson is about getting from "something went wrong" to the exact lines that say what, in a few commands.

The practice server

The practice files (opens in a new tab) are what a small online bookshop's server left behind. Nothing actually runs; you're looking at the evidence afterwards. The setup it describes is simple:

visitor  →  web server (port 80)  →  books-api app  →  database (db.internal:5432)

Visitors talk to the web server, which passes their requests to the app, which looks things up in the database. Everything is made up, including the IP addresses, which come from ranges set aside for documentation.

You downloaded them at the end of Lesson 2. (Skipped that? On your own computer, run git clone https://github.com/Kwamib/linux-practice.git first.) On your own computer, go into the folder and start a container with the files inside:

cd linux-practice
docker run -it --rm --name lesson3 -v "$PWD":/practice -w /practice ubuntu:24.04 bash

The new part is -v "$PWD":/practice. It makes the folder you're in appear inside the container at /practice. It's a live link, not a copy, so anything you change in /practice changes the real files too. -w /practice starts you in that folder. Your prompt changes to root@...:/practice#: you're inside. Everything in this lesson runs here, except one command in "Watching a log live" that says otherwise.

Look around first

Before opening anything, get the lay of the land:

pwd
ls -la
ls -l logs config notes data
ls -R config

/practice
total 8
drwxr-xr-x  8 root root  256 Oct 29 17:29 .
drwxr-xr-x  1 root root 4096 Oct 29 17:29 ..
drwxr-xr-x 12 root root  384 Oct 29 16:54 .git
-rw-------  1 root root  442 Oct 29 12:30 README.md
drwx------  4 root root  128 Oct 29 12:30 config
drwx------  3 root root   96 Oct 29 13:08 data
drwx------  6 root root  192 Oct 29 12:39 logs
drwx------  4 root root  128 Oct 29 12:30 notes
config:
total 0
drwx------ 4 root root 128 Oct 29 12:39 app
drwx------ 4 root root 128 Oct 29 12:39 web

data:
total 0
drwx------ 3 root root 96 Oct 29 12:30 uploads

logs:
total 2812
-rw------- 1 root root  322825 Oct 29 12:39 access.log
-rw------- 1 root root   33063 Oct 29 12:39 app.log
-rw------- 1 root root   15041 Oct 29 12:30 app.log.1
-rw------- 1 root root 2500077 Oct 29 12:39 debug.log

notes:
total 8
-rw------- 1 root root  80 Oct 29 12:30 contacts.txt
-rw------- 1 root root 127 Oct 29 12:30 todo.txt
config:
app  web

config/app:
app.conf  override.conf

config/web:
README.txt  site.conf

(The dates and permissions on your copy will differ.)

Giving ls several folders lists each one under its own heading. -R (recursive) goes all the way down, folder by folder. Already, the shape of the evidence is clear:

One file stands out before you've read a word: debug.log is 2,500,077 bytes, several times bigger than the other logs put together. Keep that in mind.

A useful variation: ls -ltr sorts by time (-t) and reverses it (-r), so the most recently changed files come last, right above your prompt. "What changed most recently?" is often the first question when something breaks.

Reading files

Four ways to read, depending on how much you want:

cat notes/todo.txt
head -3 logs/access.log
tail -3 logs/app.log
wc -l logs/*

Check why the API returned errors around 2pm.
Find out who is hammering the login page.
Rotate the logs before the disk fills.
203.0.113.24 - - [28/Oct/2026:00:00:55 +0000] "GET / HTTP/1.1" 200 4806 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0 Safari/537.36"
203.0.113.23 - - [28/Oct/2026:00:01:57 +0000] "GET /static/app.js HTTP/1.1" 200 1724 "-" "curl/8.5.0"
203.0.113.37 - - [28/Oct/2026:00:02:17 +0000] "GET /api/books HTTP/1.1" 200 734 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0 Safari/537.36"
2026-10-28 22:20:06 INFO  user session ended
2026-10-28 22:22:18 INFO  user session started
2026-10-28 22:23:22 INFO  user session started
   2300 logs/access.log
    638 logs/app.log
    300 logs/app.log.1
  37299 logs/debug.log
  40537 total

• cat prints a whole file. Fine for a short note; useless for 37,000 lines.
• head -3 shows the first 3 lines, and tail -3 the last 3. Without a number, they show 10. Logs are written in time order, so tail shows the most recent events.
• wc -l counts lines (wc is short for word count). logs/* means every file in logs.

The to-do list is your brief: errors around 2pm, and someone hammering the login page. The rest of this lesson answers both.

Reading a log line

Each line of the web server's log follows the same pattern. Taking the first one apart:

That fixed layout is what makes logs searchable. Lesson 4 uses it to pull out single columns and count them.

Paging with less

For anything long, use less. It shows one screen at a time and lets you move and search without loading the whole file:

less logs/access.log

less is the right tool when you want to read around something. When you want to find every line that matches, grep is faster.

Searching inside files with grep

grep prints every line that contains a pattern. It's the command you'll use most when something breaks.

grep -c ERROR logs/app.log
grep -c error logs/app.log
grep -ci error logs/app.log
grep ERROR logs/app.log | head -3

37
0
37
2026-10-28 14:00:37 ERROR database connection refused: db.internal:5432
2026-10-28 14:01:03 ERROR database connection refused: db.internal:5432
2026-10-28 14:01:28 ERROR request failed: upstream timeout after 30s

• -c counts matching lines instead of printing them: 37 errors.
• grep is case-sensitive. Searching for error found nothing, and said nothing. -i ignores case. When you're not sure how something is written, use it.

And there's the cause of the 2pm problem: the app couldn't reach its database, starting at 14:00:37. When did it recover?

grep -n "connection restored" logs/app.log
grep -B2 -A1 "connection restored" logs/app.log
grep -v INFO logs/app.log | head -5

418:2026-10-28 14:21:03 INFO  database connection restored
2026-10-28 14:17:43 INFO  user session ended
2026-10-28 14:19:38 INFO  user session started
2026-10-28 14:21:03 INFO  database connection restored
2026-10-28 14:22:02 INFO  user session ended
2026-10-28 00:13:33 WARN  retrying cache connection
2026-10-28 00:18:48 WARN  slow query: books lookup took 1191ms
2026-10-28 01:10:40 WARN  slow query: books lookup took 1838ms
2026-10-28 01:39:03 WARN  retrying cache connection
2026-10-28 01:42:58 WARN  retrying cache connection

• -n adds the line number: it's line 418. Handy for jumping straight there in less or an editor.
• -B2 -A1 shows 2 lines Before and 1 After each match, for context. A log line on its own rarely tells the whole story.
• -v inverts the search: every line that doesn't match. Hiding the routine INFO lines leaves the warnings, and here they show slow database queries hours before the outage. Whether they're connected is exactly the kind of question troubleshooting asks.

The database came back at 14:21:03, so the outage lasted about 21 minutes. Now the web server's side, and the second item on the list:

grep -c '" 50[0-9] ' logs/access.log
grep -c 198.51.100.77 logs/access.log

28
180

The pattern '" 50[0-9] ' uses [0-9], which means "any one digit", so it matches status codes 500 to 509: server errors. Visitors got 28 of them. The quotes and spaces in the pattern make sure it matches the status code, not a 50 somewhere else in the line.

And one address, 198.51.100.77, appears 180 times. Look at one of its lines with grep -m1 198.51.100.77 logs/access.log and you'll see POST /login and status 401: a failed login, 180 times in half an hour. That's the hammering.

Searching a whole folder

-r searches every file in a folder, and -l lists only the names of the files that match:

grep -r "port" config
grep -rl 8080 .

config/app/override.conf:port = 9090
config/app/app.conf:port = 8080
config/app/app.conf:database_port = 5432
config/web/README.txt:The web server forwards /api/ requests to the app on port 8080.
./config/app/app.conf
./config/web/site.conf
./config/web/README.txt
./logs/debug.log
./logs/access.log

The first search found a trap. The app's port is set twice: 8080 in app.conf, and 9090 in override.conf, which is loaded afterwards and wins. Meanwhile, the web server forwards requests to 8080. That mismatch hasn't caused anything in these logs, but it's exactly the kind of thing that does. A later lesson comes back to it.

The second search shows something else: grep matched debug.log and access.log, which have nothing to do with ports. Looking at the lines:

203.0.113.37 - - [28/Oct/2026:11:35:46 +0000] "GET /static/app.js HTTP/1.1" 200 8080 "-" ...
2026-10-28 01:00:31.281 DEBUG request id=88080b path=/api/books/311 took=52ms

In the first, 8080 is the size of a response. In the second, it's part of a random request ID, 88080b. grep finds characters, not meaning. -w matches whole words only, which rules out 88080b but not the response size, because that really is the word 8080. A match is a lead, not proof: always look at the lines.

Try patterns and options yourself, on real lines from the practice logs:

grep 'ERROR'

Finding files with find

grep searches inside files. find finds the files themselves, by name, type, size or age, anywhere under a folder:

find . -name "*.conf"
find . -name "*.jpg" | wc -l
find . -size +1M
find . -size +1M -exec ls -lh {} \;

./config/app/override.conf
./config/app/app.conf
./config/web/site.conf
12
./logs/debug.log
-rw------- 1 root root 2.4M Oct 29 12:39 ./logs/debug.log

• . is where to start: this folder, and everything under it.
• -name "*.conf" matches names ending in .conf. The * means "anything".
• -size +1M matches files bigger than one megabyte. Among 24 files, it went straight to debug.log. On a full disk, the same command run over a whole server finds the culprit in seconds.
• -exec ls -lh {} \; runs a command on each file found. {} stands for the file, and \; ends the command.

debug.log is huge because someone left debug logging switched on. Remember override.conf? It also says log_level = debug. Settings, logs and the size of the disk are all connected, which is why the to-do list's third item, rotating the logs, matters. Lesson 10 deals with full disks.

By type, and by age

find . -type d | sort

-type d finds directories, and -type f finds ordinary files. But run that one and the output is swamped by dozens of folders like ./.git/objects/1a: find searched Git's hidden folder too. To leave a folder out:

find . -not -path './.git*' -type d | sort

.
./config
./config/app
./config/web
./data
./data/uploads
./data/uploads/2026
./data/uploads/2026/09
./data/uploads/2026/10
./logs
./notes

Finding old files is a classic cleanup job. The practice photos all have today's date, so first set September's back to September with touch -d:

touch -d "2026-09-01" data/uploads/2026/09/*.jpg
find data -type f -mtime +30

data/uploads/2026/09/photo-3.jpg
data/uploads/2026/09/photo-2.jpg
data/uploads/2026/09/photo-1.jpg
data/uploads/2026/09/photo-5.jpg
data/uploads/2026/09/photo-4.jpg
data/uploads/2026/09/photo-6.jpg

-mtime +30 means changed more than 30 days ago. Notice the order: 3, 2, 1, 5. find lists files in whatever order the disk keeps them; add | sort when the order matters.

Watching a log live

Everything so far has looked at the past. When you're fixing something, you often want to watch what happens now. tail -f (follow) shows the end of a file, then waits and prints each new line the moment it's written.

Make a copy to experiment on, so the real log stays as it is, and follow it:

cp logs/app.log /tmp/live.log
tail -f /tmp/live.log

Then, in a second terminal window on your own computer, add lines to that file from outside the container. docker exec runs a command inside a container that's already running, found by its name:

docker exec lesson3 sh -c 'echo "2026-10-28 16:40:00 ERROR test line one" >> /tmp/live.log'
docker exec lesson3 sh -c 'echo "2026-10-28 16:40:05 INFO  test line two" >> /tmp/live.log'

2026-10-28 22:17:10 WARN  retrying cache connection
2026-10-28 22:20:06 INFO  user session ended
2026-10-28 22:22:18 INFO  user session started
2026-10-28 22:23:22 INFO  user session started
2026-10-28 16:40:00 ERROR test line one
2026-10-28 16:40:05 INFO  test line two

(The first six of the ten existing lines are left out.)

The two test lines appeared the instant each command ran. Press Ctrl+C to stop following. This is how you watch a server while you restart something or reproduce a problem: one window doing, one window following the log.

Break it on purpose

grep port config
grep ERROR logs/missing.log
grep -q "panic" logs/app.log; echo "exit code $?"
grep -q "ERROR" logs/app.log; echo "exit code $?"
find . -name *.log
cd logs
find . -name *.log
cd ..

grep: config: Is a directory
grep: logs/missing.log: No such file or directory
exit code 1
exit code 0
./logs/debug.log
./logs/access.log
./logs/app.log
find: paths must precede expression: `app.log'
find: possible unquoted pattern after predicate `-name'?

• Is a directory: without -r, grep searches files, not folders.
• No such file or directory: the path is wrong, and the error names it.
• Exit codes. -q (quiet) prints nothing; grep answers only through its exit code, the number every command hands back when it finishes. 0 means it found a match, 1 means it found none, and 2 means something went wrong. echo $? prints the last one. Scripts rely on this: "did the log mention ERROR?" becomes one line.
• The same find worked once and failed once. Before running a command, the shell expands an unquoted * into matching file names in the current folder. In /practice there are no .log files at the top, so *.log was passed through unchanged and it worked, by luck. Inside logs, the shell turned it into access.log app.log debug.log, and find received nonsense. Quote every pattern you give find: find . -name "*.log" works anywhere.

One more detail from that output: app.log.1 wasn't listed, because its name doesn't end in .log. To include rotated logs, use -name "*.log*".

Spot the problem

Work out what's wrong before you read the explanation.

The cleanup script that broke. A script runs find . -name *.tmp -delete. It works for weeks, then one day fails with "paths must precede expression". What changed? Someone left a .tmp file in the folder the script runs from, so the shell expanded *.tmp before find saw it. Quoting the pattern fixes it for good.

Where you are now

You answered the to-do list from the evidence alone: a 21-minute database outage behind 28 server errors, and 180 failed logins from one address. Lesson 4 goes from finding lines to working with them: pulling out columns, counting, sorting, and answering questions like "which hour had the most errors?" in one line.

Check yourself

Key terms

• Log: a file a program writes to, one line per event, in time order.
• Pattern: what grep searches for. [0-9] means any digit, * in find means anything.
• Recursive: going into every folder inside a folder, all the way down.
• Exit code: the number a command returns when it finishes; 0 means success.
• Glob: a pattern like *.log that the shell expands into matching file names.
• Bind mount: a folder on your computer made to appear inside a container, with -v.

Go deeper

• The practice files (opens in a new tab): try grep -c " 404 " logs/access.log, then work out which pages people asked for that didn't exist.
• Which hour of the day had the most server errors? You can get close with grep alone. Lesson 4 does it properly in one line.
• explainshell (opens in a new tab): paste any find or grep command and see what every part means.
• Lesson 2, The shell, for paths and ls -l.