Processes and signals

The problem: nothing is running

Lesson 5 gave the bookshop its settings, its logs and its own user. But nothing is actually running yet. A program sitting on disk does nothing; a program that's running is a process, and a server is really just a set of processes. This lesson starts the app, watches it, and stops it, in all the ways you'll need when something misbehaves.

On your own computer, from the folder that holds linux-practice:

cd linux-practice && docker run -it --rm --name lesson6 -v "$PWD":/practice -w /practice ubuntu:24.04 bash

Everything else runs inside the container. The container starts fresh, so first put back the two pieces of Lesson 5 this lesson needs, the bookshop user and its log folder:

useradd --system --shell /usr/sbin/nologin bookshop
mkdir -p /var/log/books-api
chown bookshop:bookshop /var/log/books-api

The app

The real bookshop app isn't in the practice files, so make a stand-in. It writes a log line every 2 seconds, forever, like a server handling requests. cat > file <<'EOF' writes every line you type into the file, until a line that says just EOF:

cat > /usr/local/bin/books-api <<'EOF'
#!/bin/bash
trap 'echo "$(date "+%F %T") INFO  shutting down"; exit 0' TERM
echo "$(date '+%F %T') INFO  books-api started, process $$"
while true; do
  echo "$(date '+%F %T') INFO  request handled"
  sleep 2
done
EOF
chmod +x /usr/local/bin/books-api

What each part does, in plain words:

• trap '...' TERM says: when someone asks this program to stop (the TERM signal, coming up below), write shutting down and exit. Real apps do the same, to finish what they're doing and close things properly.
• $$ is the program's own process number.
• while true; do ... done repeats forever: write a line, wait 2 seconds, again. Lesson 12 covers scripts properly.

What's running: ps

ps
ps -ef

    PID TTY          TIME CMD
      1 pts/0    00:00:00 bash
     14 pts/0    00:00:00 ps
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 19:40 pts/0    00:00:00 bash
root          15       1  0 19:40 pts/0    00:00:00 ps -ef

(Your times and numbers will differ.) Plain ps shows the processes started from your terminal. -ef shows every process on the machine, in full. In a container, that's a very short list: just your shell and ps itself.

Every process has a PID (process ID), a number unique while it runs. Every process except the first was started by another one, its parent, whose number is the PPID. Process 1 is special: on a real server it's systemd (Lesson 7), which starts everything else. In a container it's whatever the container was told to run, here bash. When this bash exits, the container stops, which is why exit ends your session.

Start the app

Run it as the bookshop user, with its output going to its log, and with & at the end, which means "run it in the background and give me my prompt back":

su -s /bin/bash bookshop -c 'books-api >> /var/log/books-api/app.log 2>&1 &'
ps -ef

UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 19:40 pts/0    00:00:00 bash
bookshop      18       1  0 19:40 ?        00:00:00 /bin/bash /usr/local/bin/boo
bookshop      21      18  0 19:40 ?        00:00:00 sleep 2
root          22       1  0 19:40 pts/0    00:00:00 ps -ef

Two new processes, both belonging to bookshop. Process 18 is the app (the name is cut off to fit). Process 21 is the sleep 2 it's waiting on, and its PPID, 18, says the app started it. The app's own PPID is 1: su started it and exited, and a process whose parent goes away is adopted by process 1. Its TTY is ?: it isn't attached to any terminal, which is normal for a server program.

Find it by name with pgrep, and choose the columns you want with ps -o:

pgrep -a books-api
ps -o pid,ppid,user,stat,etime,cmd -u bookshop

18 /bin/bash /usr/local/bin/books-api
    PID    PPID USER     STAT     ELAPSED CMD
     18       1 bookshop S          00:02 /bin/bash /usr/local/bin/books-api
     25      18 bookshop S          00:00 sleep 2

pgrep prints the PID of each match, and -a adds the full command. -u bookshop shows only that user's processes. ELAPSED is how long each has been running: the app 2 seconds, and a fresh sleep 2 every two seconds, which is why its number keeps changing.

Watch it work, the way Lesson 3 did:

tail -f /var/log/books-api/app.log

2026-10-29 19:40:34 INFO  books-api started, process 18
2026-10-29 19:40:34 INFO  request handled
2026-10-29 19:40:36 INFO  request handled
2026-10-29 19:40:38 INFO  request handled
2026-10-29 19:40:40 INFO  request handled

A new line every 2 seconds. Press Ctrl+C to stop watching; the app keeps running.

Inside a process: /proc

Lesson 2 found numbered folders in /proc, one per process. Now you know what they are. $(pgrep books-api) is replaced by the app's PID before the command runs:

grep -E '^(Name|State|PPid|Uid)' /proc/$(pgrep books-api)/status
tr '\0' ' ' < /proc/$(pgrep books-api)/cmdline; echo

Name:	books-api
State:	S (sleeping)
PPid:	1
Uid:	999	999	999	999
/bin/bash /usr/local/bin/books-api 

Everything ps shows comes from here. Uid: 999 is bookshop from Lesson 5. cmdline is the exact command that started it, with its pieces separated by an invisible character, which tr turns into spaces.

Jobs: background and foreground

& works for anything you run. The shell numbers each background command as a job:

sleep 300 &
sleep 600 &
jobs

[1] 42
[2] 45
[1]-  Running                 sleep 300 &
[2]+  Running                 sleep 600 &

[1] 42 means job 1, process 42. jobs lists them; + marks the one most commands act on by default. Now bring job 2 to the front, so it has your terminal, and press Ctrl+Z:

fg %2

sleep 600
^Z
[2]+  Stopped                 sleep 600

fg (foreground) brought it to the front, and your prompt disappeared while it ran. Ctrl+Z paused it and gave your prompt back. It's still in memory, doing nothing. bg (background) lets it carry on, behind the scenes:

bg %2
jobs
kill %1 %2

[2]+ sleep 600 &
[1]-  Running                 sleep 300 &
[2]+  Running                 sleep 600 &
[1]-  Terminated              sleep 300
[2]+  Terminated              sleep 600

kill %1 %2 stopped both, by job number. This is the everyday use: start something slow with &, or press Ctrl+Z on something you forgot to background, then bg it. Jobs belong to the shell you started them in. The app isn't one of your jobs, because su started it, so for the app you use its PID.

Signals: asking a process to stop

kill doesn't really kill. It sends a signal, a short message from the kernel to a process. There are dozens; a handful matter:

Ask the app to stop the polite way:

kill $(pgrep books-api)
sleep 3; tail -3 /var/log/books-api/app.log
pgrep -a books-api

2026-10-29 19:40:56 INFO  request handled
2026-10-29 19:40:58 INFO  request handled
2026-10-29 19:41:00 INFO  shutting down

The app caught TERM, wrote its last line, and exited, so pgrep found nothing. Run pgrep straight after the kill, though, and the app is still there for up to 2 seconds: bash finishes the sleep it's in before it runs the trap. Real apps do the same, finishing the request they're on, which is why "stop" can take a moment. Be patient before reaching for -9.

When TERM isn't enough. A program can also ignore TERM. Here's one that does:

bash -c 'trap "" TERM; while true; do sleep 1; done' &
kill %1
sleep 1; jobs
kill -9 %1
sleep 1; jobs

[1] 62
[1]+  Running                 bash -c 'trap "" TERM; while true; do sleep 1; done' &
[1]+  Killed                  bash -c 'trap "" TERM; while true; do sleep 1; done'

TERM did nothing. kill -9 sends KILL, which never reaches the program at all: the kernel just removes it. That always works, which is exactly why it's dangerous, as "Break it on purpose" shows.

Exit codes tell you how it ended. A process stopped by a signal exits with 128 plus the signal's number:

sleep 300 &
kill %1; wait %1; echo "exit code $?"
sleep 300 &
kill -9 %1; wait %1; echo "exit code $?"

[1] 70
[1]+  Terminated              sleep 300
exit code 143
[1] 71
[1]+  Killed                  sleep 300
exit code 137

143 is 128 + 15 (TERM), and 137 is 128 + 9 (KILL). You've met 137 before: in the Docker track, a container killed for using too much memory exits with 137, because the kernel sent it KILL. wait waits for a job to finish, so $? holds its exit code.

Try each signal on the app:

Pause and carry on. Ctrl+Z works only on your own jobs, but kill -STOP pauses any process:

su -s /bin/bash bookshop -c 'books-api >> /var/log/books-api/app.log 2>&1 &'
kill -STOP $(pgrep books-api)
ps -o pid,user,stat,cmd -C books-api
sleep 4; tail -1 /var/log/books-api/app.log; date '+%T'
kill -CONT $(pgrep books-api)

    PID USER     STAT CMD
     14 bookshop T    /bin/bash /usr/local/bin/books-api
2026-10-29 19:42:20 INFO  request handled
19:42:26

STAT T: stopped. The last log line is 6 seconds old, so nothing has been written while it was paused. -C books-api picks processes by command name. After kill -CONT, it carries on writing every 2 seconds as if nothing happened.

Live view: top

ps is a snapshot. top keeps updating, busiest first. Run plain top to see it live, and press q to quit. Here's one snapshot of it, with -b -n 1 (batch mode, once), which also works inside scripts:

top -b -n 1 | head -12

top - 19:41:21 up 4 min,  0 user,  load average: 0.04, 0.01, 0.00
Tasks:   5 total,   1 running,   4 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   8027.6 total,   7541.9 free,    445.9 used,    254.8 buff/cache     
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   7581.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0    5112   4544   3420 S   0.0   0.1   0:00.02 bash
     74 bookshop  20   0    4332   3304   3052 S   0.0   0.0   0:00.00 books-a+
     77 bookshop  20   0    2704   1712   1620 S   0.0   0.0   0:00.00 sleep
     78 root      20   0    8724   5068   3012 R   0.0   0.1   0:00.00 top
     79 root      20   0    2716   1664   1568 S   0.0   0.0   0:00.00 head

(Your memory figures are your machine's, and will differ.) The top five lines are the whole machine:

• load average is how many processes, on average, were running or waiting to run over the last 1, 5 and 15 minutes. Compare it with the number of processors: a load of 4 on a 4-processor machine is busy but fine; 20 is a queue.
• %Cpu(s): us is time spent on programs, sy on the kernel, id idle, and wa waiting for the disk. High wa points at storage, not the program.
• MiB Mem: avail Mem is the one that matters, how much programs could still use. Lesson 10 covers memory.

Below that, one line per process: RES is the memory it's really using, in kilobytes, and %CPU and %MEM its share. In live top, M sorts by memory, P by processor, and k asks for a PID to kill. For a one-off answer, ps can sort too:

ps aux --sort=-%mem | head -4

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root          24  0.0  0.0   7640  3656 pts/0    R+   21:42   0:00 ps aux --sort=-%mem
root           1  0.1  0.0   4308  3644 pts/0    Ss   21:42   0:00 bash
bookshop      23  0.0  0.0   4472  3052 ?        S    21:42   0:00 /bin/bash /usr/local/bin/books-api

aux is the other classic form of "every process", with memory columns. --sort=-%mem puts the biggest user of memory first. On a real server, that line answers "what's eating the memory?" in one go.

Surviving logout: nohup

When you log out, or your connection drops, the terminal closes and sends HUP to the jobs you started from it, and by default HUP stops them. nohup runs a command with HUP ignored, so it keeps going:

cd /tmp
nohup sleep 120 &

[1] 37
nohup: ignoring input and appending output to 'nohup.out'

Its output goes to a file called nohup.out, since there may be no terminal left to show it. Give it your own file instead, with Lesson 4's redirection: nohup command > job.log 2>&1 &. It's fine for a one-off long job. For anything that should always be running, it's the wrong tool, and Lesson 7 shows the right one. Run cd /practice to go back.

Break it on purpose

Kill by name, hit the wrong thing. Start a sleep of your own, then stop "all the sleeps":

sleep 900 &
pgrep -a sleep
pkill sleep
sleep 1; pgrep -a sleep

[1] 86
85 sleep 2
86 sleep 900
[1]+  Terminated              sleep 900
96 sleep 2

There were two: yours, and the one inside the app. pkill stopped both. The app survived this time, because its sleep just ended early, but on a real server "kill everything called python" or "everything called java" can take out a service you didn't mean to touch. Run pgrep -a first, and look at what matches. pkill -x matches the whole name exactly, and pkill -u bookshop limits it to one user.

kill -9 skips the goodbye.

kill -9 $(pgrep books-api)
tail -2 /var/log/books-api/app.log

2026-10-29 19:41:27 INFO  request handled
2026-10-29 19:41:29 INFO  request handled

No shutting down line. The app never got the chance: KILL doesn't ask. Here, that's one missing log line. In a real app, it can be a half-written file, a database transaction left open, or a lock file that stops the next start. Send TERM, give it time, and use -9 only when TERM hasn't worked.

Another user's process. Start the app again, make Sam from Lesson 5, and have Sam try to stop it:

useradd --create-home --shell /bin/bash sam
su -s /bin/bash bookshop -c 'books-api >> /var/log/books-api/app.log 2>&1 &'
su - sam -c "kill $(pgrep books-api)"

-bash: line 1: kill: (112) - Operation not permitted

You can only signal your own processes; root can signal anyone's. The same rule as Lesson 5's files, applied to running programs. The double quotes matter here: they let $(pgrep books-api) run first, as root, so Sam's kill gets a number.

A PID that's gone. Save the app's PID, stop it, then try again:

OLD=$(pgrep books-api); kill $OLD; sleep 3
kill $OLD

bash: kill: (112) - No such process

PIDs aren't forever. Once a process ends, its number means nothing, and the system will eventually hand it to a new process. A script that saves a PID and uses it much later can, in the worst case, signal something else entirely. Always look the PID up again right before you use it.

Spot the problem

Work out what's wrong before you read the explanation.

Twice as many requests. After a restart, the app's log suddenly has a line every second instead of every two. Nothing in the app changed. What happened?

su -s /bin/bash bookshop -c 'books-api >> /var/log/books-api/app.log 2>&1 &'
su -s /bin/bash bookshop -c 'books-api >> /var/log/books-api/app.log 2>&1 &'
sleep 3; pgrep -a books-api; tail -4 /var/log/books-api/app.log

128 /bin/bash /usr/local/bin/books-api
134 /bin/bash /usr/local/bin/books-api
2026-10-29 19:41:42 INFO  request handled
2026-10-29 19:41:43 INFO  request handled
2026-10-29 19:41:44 INFO  request handled
2026-10-29 19:41:45 INFO  request handled

Someone started it again without stopping the old one first. Two copies are running, both writing to the same log. pgrep -a shows it in a second. For an app that listens on a port, the second copy fails instead, with "address already in use" (Lesson 9). Either way, "is it already running?" is the first question before starting anything by hand. Stop both with pkill -x books-api, and check with pgrep -c books-api, which counts matches.

Where you are now

The app runs, as its own user, and you can find it, watch it, pause it and stop it, politely or not. But everything here was by hand. If the app crashes, nothing restarts it. If the server reboots, nothing starts it. And nothing stops you starting it twice. Lesson 7 hands all of that to systemd, the service manager that runs as process 1 on real servers, and adds journalctl for reading what services say.

Check yourself

Key terms

• Process: a running program, with its own PID.
• PID and PPID: a process's ID, and its parent's.
• Process 1: the first process, which starts everything else. systemd on servers; the container's command in a container.
• Job: a command started from your shell, with a job number like %1.
• Foreground and background: in front of your terminal, holding the prompt, or behind it, with &.
• Signal: a short message to a process, like TERM (please stop) or KILL (stop now).
• Load average: how many processes were running or waiting, on average, over the last 1, 5 and 15 minutes.

Go deeper

• Run kill -l to list every signal by name and number.
• While the app runs, ls -l /proc/$(pgrep books-api)/fd shows the files it has open. Its log is there, as number 1, its standard output from Lesson 4.
• Run top and press h for every key it understands.
• Lesson 5, Permissions and users, for the bookshop user and why the app runs as it.