Services and logs with systemd

The problem: started by hand isn't enough

Lesson 6 ended with three problems. If the app crashes, nothing restarts it. If the server reboots, nothing starts it. And nothing stops you starting it twice. On a real server, one program solves all three: the service manager. On almost every Linux distribution today, that's systemd. It runs as process 1, starts everything else at boot, restarts what crashes, and collects what every service says into one log, called the journal.

From a container to a server

There's a catch. Containers are made to run one program, not a whole system with a service manager. When systemd was started in an ordinary container while preparing this lesson, it stopped at once with exit code 255. It needs control groups (cgroups), the kernel feature it uses to track and limit each service, and a normal container isn't allowed to manage them. There are ways around it, but from here on, the track needs things a container keeps getting in the way of: services now, then real networking, disks and SSH.

So the bookshop moves onto a virtual machine (VM): a whole computer, simulated by your own, with its own kernel, its own disk and its own network. It boots like a real server, and systemd runs it, as process 1. A container shares your computer's kernel and starts in a second; a VM has its own kernel and takes a minute to boot. Lessons 1 to 6 needed the first. Lessons 7 to 15 need the second.

What Vagrant is

You could make a VM by hand: open Fusion or VirtualBox, pick an Ubuntu image, set the memory, set up its network, install SSH, share a folder. That works once. It's slow, easy to get slightly wrong, and impossible to hand to someone else. Vagrant writes all of it down in one file, the Vagrantfile, so vagrant up builds the same machine every time, on anyone's computer, and vagrant destroy throws it away without a second thought.

It has three pieces:

• The Vagrantfile: what the machine should be.
• A box: the ready-made image it starts from, here Ubuntu 24.04.
• A provider: the program that actually runs the VM, such as VMware Fusion or VirtualBox. Vagrant itself runs nothing; it tells the provider what to build.

Vagrant is made by HashiCorp, the company behind Terraform, and the idea is the same: describe the machine in a file, and let the tool build it. Terraform does that for servers in the cloud; Vagrant does it for machines on your own computer. It's a tool for practice and development machines, so a whole team can test on identical ones, not for running production servers. The Ansible track uses it again, to give Ansible machines to set up.

The practice files have a Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "bento/ubuntu-24.04"
  config.vm.hostname = "bookshop"
  config.vm.synced_folder ".", "/practice"
  config.vm.synced_folder ".", "/vagrant", disabled: true

  config.vm.provider "vmware_desktop" do |v|
    v.memory = 2048
    v.cpus = 2
    v.vmx["ethernet0.pcislotnumber"] = "160"
  end

  config.vm.provider "virtualbox" do |v|
    v.memory = 2048
    v.cpus = 2
  end
end

It says: start from a ready-made Ubuntu 24.04 image (a box), call the machine bookshop, share this folder into it at /practice as -v did for Docker, and give it 2 GB of memory and 2 processors. The two provider sections cover the two programs that can run the VM; the vmx line is a network setting Fusion needs for this box.

Install Vagrant, once

Vagrant needs a program that actually runs VMs, called a provider:

• On a Mac with Apple Silicon: VMware Fusion, which is free for personal use. Then brew install --cask vagrant-vmware-utility and vagrant plugin install vagrant-vmware-desktop, which let Vagrant drive Fusion.
• On Windows, or a Mac or Linux machine with an Intel or AMD processor: VirtualBox, which is free, and needs nothing extra.

Then install Vagrant itself from vagrantup.com (opens in a new tab), or with brew install hashicorp/tap/hashicorp-vagrant on a Mac, and check it with vagrant --version.

Start the server

On your own computer, in your practice folder. Lesson 2 cloned it; if you cloned it before the Vagrantfile was added, run git pull in it first:

cd linux-practice
vagrant up

The first time, Vagrant downloads the Ubuntu box, about 1 GB, then boots it. The end of its output says the server is ready and the folder is shared:

==> default: Machine booted and ready!
==> default: Setting hostname...
==> default: Enabling and configuring shared folders...
    default: -- /Users/you/linux-practice: /practice

Log in to it:

vagrant ssh

Ubuntu greets you with a summary of the machine, and your prompt changes to vagrant@bookshop:~$. You're on the server. vagrant ssh used SSH, which Lesson 11 covers, with a key Vagrant made for you.

Three prompts now. Every vagrant command runs on your own computer. Everything else in this lesson runs on the server. Type vagrant reload on the server and you get -bash: vagrant: command not found, because the server doesn't have Vagrant; your computer does.

Look around: a real server, and sudo

whoami
id
sudo whoami
ps -p 1 -o pid,comm
systemctl is-system-running

vagrant
uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),101(lxd)
root
    PID COMMAND
      1 systemd
running

For the first time in the track, you're not root. You're vagrant, an ordinary user, the way you'd log in to a real cloud server as ubuntu. Lesson 5 promised sudo: here it is. sudo command runs one command as root, and it's allowed because vagrant is in the sudo group (27(sudo) in the list). The adm group lets it read system logs.

Process 1 is systemd, not bash. And systemctl is-system-running says running: everything systemd was asked to start, started.

sudo and > don't mix. You'll need to write files as root in a moment, and the obvious way fails:

sudo echo "test" > /etc/books-test.txt
echo "test" | sudo tee /etc/books-test.txt
sudo rm /etc/books-test.txt

-bash: /etc/books-test.txt: Permission denied
test

sudo ran echo as root, but the > belongs to your shell, which set it up first, as vagrant (Lesson 4: the shell handles redirection before the command runs). tee from Lesson 4 fixes it: the writing is done by tee, and tee is what runs as root. sudo tee file > /dev/null is the standard way to write a file as root, with > /dev/null hiding the copy tee prints.

Make the app a service

Put back the app's user from Lesson 5, and the app from Lesson 6, with one change: it no longer writes to a log file. It just prints. Under systemd, whatever a service prints goes to the journal.

sudo useradd --system --shell /usr/sbin/nologin bookshop
sudo tee /usr/local/bin/books-api > /dev/null <<'EOF'
#!/bin/bash
trap 'echo "INFO  shutting down"; exit 0' TERM
echo "INFO  books-api started, process $$"
while true; do
  echo "INFO  request handled"
  sleep 2
done
EOF
sudo chmod +x /usr/local/bin/books-api

The dates are gone from the messages too, because the journal adds its own.

Now the unit file: a short text file that tells systemd what the service is and how to run it. Services you add go in /etc/systemd/system/, with a name ending in .service:

sudo tee /etc/systemd/system/books-api.service > /dev/null <<'EOF'
[Unit]
Description=Bookshop API (practice)
After=network.target

[Service]
User=bookshop
ExecStart=/usr/local/bin/books-api
Restart=on-failure
RestartSec=3

[Install]
WantedBy=multi-user.target
EOF

systemd reads unit files when it starts, so tell it to read them again, then ask about the new service:

sudo systemctl daemon-reload
systemctl status books-api --no-pager

○ books-api.service - Bookshop API (practice)
     Loaded: loaded (/etc/systemd/system/books-api.service; disabled; preset: enabled)
     Active: inactive (dead)

systemd knows about it (loaded), it isn't set to start at boot (disabled), and it isn't running (inactive). --no-pager prints straight to the screen instead of opening the output in less.

Start it and look

sudo systemctl start books-api
sleep 5
systemctl status books-api --no-pager

● books-api.service - Bookshop API (practice)
     Loaded: loaded (/etc/systemd/system/books-api.service; disabled; preset: enabled)
     Active: active (running) since Thu 2026-10-29 22:12:57 UTC; 5s ago
   Main PID: 2546 (books-api)
      Tasks: 2 (limit: 2211)
     Memory: 584.0K (peak: 896.0K)
        CPU: 3ms
     CGroup: /system.slice/books-api.service
             ├─2546 /bin/bash /usr/local/bin/books-api
             └─2550 sleep 2

Oct 29 22:12:57 bookshop systemd[1]: Started books-api.service - Bookshop API (practice).
Oct 29 22:12:57 bookshop books-api[2546]: INFO  books-api started, process 2546
Oct 29 22:12:57 bookshop books-api[2546]: INFO  request handled
Oct 29 22:12:59 bookshop books-api[2546]: INFO  request handled
Oct 29 22:13:01 bookshop books-api[2546]: INFO  request handled

(Your times and numbers will differ.) Everything Lesson 6 dug out by hand, in one place:

• Active: running, and for how long. The dot at the start turns from ○ to ●.
• Main PID: the app's process number, which systemctl show -p MainPID --value books-api also prints on its own.
• Memory and CPU: what it's using.
• CGroup: every process that belongs to the service, here the app and its sleep. This is what systemd needed control groups for: it always knows which processes are part of a service, so stopping the service stops all of them.
• The last lines are the newest entries from its journal.

The journal is read with journalctl. -u picks one service (unit), and -n 5 the last five lines:

journalctl -u books-api --no-pager -n 5
ps -o pid,ppid,user,cmd -u bookshop

Oct 29 22:12:57 bookshop systemd[1]: Started books-api.service - Bookshop API (practice).
Oct 29 22:12:57 bookshop books-api[2546]: INFO  books-api started, process 2546
Oct 29 22:12:57 bookshop books-api[2546]: INFO  request handled
Oct 29 22:12:59 bookshop books-api[2546]: INFO  request handled
Oct 29 22:13:01 bookshop books-api[2546]: INFO  request handled
    PID    PPID USER     CMD
   2546       1 bookshop /bin/bash /usr/local/bin/books-api
   2550    2546 bookshop sleep 2

The app only printed INFO request handled. The journal added the time, the machine's name, and which program said it, books-api[2546]. Its parent is process 1, systemd, and it runs as bookshop.

Stop it, and start it at boot

sudo systemctl stop books-api
journalctl -u books-api --no-pager -n 4
systemctl is-active books-api

Oct 29 23:48:57 bookshop books-api[5464]: Terminated
Oct 29 23:48:57 bookshop books-api[5464]: INFO  shutting down
Oct 29 23:48:57 bookshop systemd[1]: books-api.service: Deactivated successfully.
Oct 29 23:48:57 bookshop systemd[1]: Stopped books-api.service - Bookshop API (practice).
inactive

stop sends TERM, so the app said shutting down, as it did in Lesson 6. But two things are different. It stopped at once, with no 2-second wait. And there's a Terminated line first. systemd sends TERM to every process in the service's control group, not just the main one, so the sleep the app was waiting on was stopped too. Bash reports that with Terminated, and with nothing left to wait for, runs its trap straight away. If a service ignores TERM, systemd waits (90 seconds, by default), then sends KILL, the same "polite, then forced" order Lesson 6 recommended.

start and stop are for now. enable is for every boot from now on: it links the unit into multi-user.target, the set of services a normal boot starts. --now also starts it, so you don't need both commands:

sudo systemctl disable books-api
sudo systemctl enable --now books-api
systemctl is-enabled books-api
ls -l /etc/systemd/system/multi-user.target.wants/ | grep books

Removed "/etc/systemd/system/multi-user.target.wants/books-api.service".
Created symlink /etc/systemd/system/multi-user.target.wants/books-api.service → /etc/systemd/system/books-api.service.
enabled
lrwxrwxrwx 1 root root 37 Oct 29 22:47 books-api.service -> /etc/systemd/system/books-api.service

(The disable is only there so you can see what enable does: it was already enabled from before.) Enabling is nothing more than a symbolic link, a file that points at another file, in the .wants folder. At boot, systemd starts everything linked there. disable removes the link. List that folder without the grep, and you'll see every other service this server starts at boot, cron and ssh among them.

Reboot the server, and see it come back by itself. This is a vagrant command, so exit to your own computer first:

exit

Then, on your own computer:

vagrant reload
vagrant ssh -c 'systemctl is-active books-api; journalctl -u books-api -b --no-pager -n 3; journalctl -u books-api -b -1 --no-pager | tail -4'

==> default: Attempting graceful shutdown of VM...
...
==> default: Machine booted and ready!
active
Oct 29 23:50:00 bookshop books-api[886]: INFO  request handled
Oct 29 23:50:02 bookshop books-api[886]: INFO  request handled
Oct 29 23:50:04 bookshop books-api[886]: INFO  request handled
Oct 29 23:49:13 bookshop books-api[5626]: Terminated
Oct 29 23:49:13 bookshop books-api[5626]: INFO  shutting down
Oct 29 23:49:13 bookshop systemd[1]: books-api.service: Deactivated successfully.
Oct 29 23:49:13 bookshop systemd[1]: Stopped books-api.service - Bookshop API (practice).

(The middle of Vagrant's output is left out.) vagrant reload shut the server down politely and booted it again. Nobody started the app, and it's active, as process 886, one of the first started at boot. -b shows this boot's journal; -b -1 shows the boot before, which ends with systemd stopping the app cleanly as the machine shut down. The journal is kept across reboots, so "what happened before it restarted?" always has an answer.

(Vagrant may also print a WARNING about a VMX setting called ethernet0.pcislotnumber with Fusion. The practice files' Vagrantfile sets it, so you shouldn't see it.)

When it crashes

Lesson 6 left the app dead after kill -9. Now, back on the server with vagrant ssh, do the same:

sudo kill -9 $(systemctl show -p MainPID --value books-api)
sleep 5
journalctl -u books-api --no-pager -n 8

Oct 29 22:50:07 bookshop books-api[3338]: INFO  request handled
Oct 29 22:50:09 bookshop books-api[3338]: INFO  request handled
Oct 29 22:50:11 bookshop systemd[1]: books-api.service: Main process exited, code=killed, status=9/KILL
Oct 29 22:50:11 bookshop systemd[1]: books-api.service: Failed with result 'signal'.
Oct 29 22:50:14 bookshop systemd[1]: books-api.service: Scheduled restart job, restart counter is at 2.
Oct 29 22:50:14 bookshop systemd[1]: Started books-api.service - Bookshop API (practice).
Oct 29 22:50:14 bookshop books-api[3447]: INFO  books-api started, process 3447
Oct 29 22:50:14 bookshop books-api[3447]: INFO  request handled

Killed at 22:50:11, restarted at 22:50:14, exactly RestartSec=3 later, as a new process, 3447. The restart counter counts how many times systemd has restarted it: 2 here, because it was killed twice while preparing this lesson. If it keeps climbing, the app is crashing over and over, and the journal lines just before each restart say why.

Now stop it cleanly, the way kill asks, and compare:

sudo kill $(systemctl show -p MainPID --value books-api)
sleep 5
systemctl is-active books-api
journalctl -u books-api --no-pager -n 3
sudo systemctl start books-api

inactive
Oct 29 23:46:54 bookshop books-api[5447]: INFO  request handled
Oct 29 23:46:56 bookshop books-api[5447]: INFO  shutting down
Oct 29 23:46:56 bookshop systemd[1]: books-api.service: Deactivated successfully.

No restart. The app caught TERM, said goodbye, and exited with code 0, which isn't a failure, so on-failure left it alone. Choose Restart=always instead, and systemd restarts it after any exit. Try the combinations:

Reading the journal

The journal holds what every service says, plus systemd's own messages and the kernel's. journalctl with the right options finds anything:

--since takes plain words. Run one:

journalctl -u books-api --since "2 minutes ago" --no-pager | tail -3
journalctl -p warning -b --no-pager | tail -5

Oct 29 22:49:37 bookshop books-api[3338]: INFO  request handled
Oct 29 22:49:39 bookshop books-api[3338]: INFO  request handled
Oct 29 22:49:41 bookshop books-api[3338]: INFO  request handled
Oct 29 22:09:39 bookshop systemd-networkd[1847]: eth0: found matching network '/run/systemd/network/10-netplan-enp2s0.network', based on potentially unpredictable interface name.
Oct 29 22:09:39 bookshop systemd-networkd[1847]: eth0: found matching network '/run/systemd/network/10-netplan-enp2s0.network', based on potentially unpredictable interface name.
Oct 29 22:09:39 bookshop systemd-networkd[1847]: eth0: found matching network '/run/systemd/network/10-netplan-enp2s0.network', based on potentially unpredictable interface name.
Oct 29 22:47:29 bookshop systemd[1]: books-api.service: Main process exited, code=killed, status=9/KILL
Oct 29 22:47:29 bookshop systemd[1]: books-api.service: Failed with result 'signal'.

-p warning is the fastest way to ask "has anything gone wrong on this machine?". It showed the app being killed, and a harmless warning from the network service, which Lesson 9 is about. Lessons 3 and 4 still apply: journalctl output pipes into grep, awk and sort like any other text.

What else is running?

systemctl list-units --type=service --state=running --no-pager | head -12
systemctl --failed --no-pager

  UNIT                      LOAD   ACTIVE SUB     DESCRIPTION
  books-api.service         loaded active running Bookshop API (practice)
  cron.service              loaded active running Regular background program processing daemon
  dbus.service              loaded active running D-Bus System Message Bus
  getty@tty1.service        loaded active running Getty on tty1
  ModemManager.service      loaded active running Modem Manager
  multipathd.service        loaded active running Device-Mapper Multipath Device Controller
  open-vm-tools.service     loaded active running Service for virtual machines hosted on VMware
  polkit.service            loaded active running Authorization Manager
  rsyslog.service           loaded active running System Logging Service
  ssh.service               loaded active running OpenBSD Secure Shell server
  systemd-journald.service  loaded active running Journal Service
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.

Your app is one service among many. ssh.service is what vagrant ssh connects to; cron.service runs scheduled jobs (Lesson 13); systemd-journald.service is the journal itself. (Your list will differ a little: open-vm-tools is there because this server runs on VMware.) systemctl --failed listing nothing is what a healthy server looks like.

Break it on purpose

Edit, and forget to reload. Misspell the program's path, and restart without telling systemd:

sudo sed -i 's|ExecStart=/usr/local/bin/books-api|ExecStart=/usr/local/bin/books-apii|' /etc/systemd/system/books-api.service
sudo systemctl restart books-api
sudo systemctl daemon-reload
sudo systemctl restart books-api
sleep 2
systemctl status books-api --no-pager | head -8

Warning: The unit file, source configuration file or drop-ins of books-api.service changed on disk. Run 'systemctl daemon-reload' to reload units.
● books-api.service - Bookshop API (practice)
     Loaded: loaded (/etc/systemd/system/books-api.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2026-10-29 23:46:35 UTC; 2s ago
    Process: 5262 ExecStart=/usr/local/bin/books-apii (code=exited, status=203/EXEC)
   Main PID: 5262 (code=exited, status=203/EXEC)
        CPU: 1ms

Two lessons in one. The first restart came with a warning: systemd still had the old file in memory, so it restarted the old, working version and told you. Editing a unit file does nothing until daemon-reload. After the reload, the restart used the new path, and failed with 203/EXEC: systemd couldn't run the program. That's a wrong path, or a file without x (Lesson 5). activating (auto-restart) means Restart=on-failure is trying again every 3 seconds, which won't help: a typo doesn't fix itself. Put it back:

sudo sed -i 's|books-apii|books-api|' /etc/systemd/system/books-api.service
sudo systemctl daemon-reload
sudo systemctl restart books-api

A user that doesn't exist.

sudo sed -i 's/^User=bookshop/User=bokshop/' /etc/systemd/system/books-api.service
sudo systemctl daemon-reload
sudo systemctl restart books-api
sleep 2
journalctl -u books-api --no-pager -n 4

Oct 29 23:46:46 bookshop systemd[1]: Started books-api.service - Bookshop API (practice).
Oct 29 23:46:46 bookshop (ooks-api)[5393]: books-api.service: Failed to determine user credentials: No such process
Oct 29 23:46:46 bookshop systemd[1]: books-api.service: Main process exited, code=exited, status=217/USER
Oct 29 23:46:46 bookshop systemd[1]: books-api.service: Failed with result 'exit-code'.

217/USER: systemd couldn't switch to the user. The message says No such process, which is misleading: it means "no such user". The lookup reuses an error code that doesn't fit. When a message doesn't make sense, the status code is the reliable part. Put it back:

sudo sed -i 's/^User=bokshop/User=bookshop/' /etc/systemd/system/books-api.service
sudo systemctl daemon-reload
sudo systemctl restart books-api

The 200-numbers are systemd's own codes for "I couldn't even start it". The ones you'll meet most:

Spot the problem

Work out what's wrong before you read the explanation.

It worked until the reboot. Someone installs a new service, runs sudo systemctl start on it, checks status (active (running)), and goes home. That night the server reboots for an update, and the service is down in the morning, with nothing in its journal for the new boot. Why? It was started, but never enabled. start is for now; enable is for every boot. status showed it, on the Loaded: line: disabled. sudo systemctl enable --now NAME does both at once, which is why it's the habit worth keeping.

Where you are now

The bookshop runs on a real server, as a real service: started at boot, restarted when it crashes, run as its own user, and logging to the journal. Lesson 8 adds the rest of the setup the practice files describe: installing software with apt, then nginx in front of the app and PostgreSQL behind it, each one a service you now know how to check.

Check yourself

Key terms

• Service manager: the program that starts, stops and watches every service. systemd, on almost every distribution.
• Virtual machine (VM): a whole computer simulated by yours, with its own kernel.
• Vagrant, box, provider: the tool that makes VMs from a Vagrantfile; the ready-made image it starts from; the program that actually runs the VM.
• Unit file: the text file that tells systemd how to run a service.
• Enabled: set to start at boot. Separate from running now.
• Journal: systemd's log of everything, read with journalctl.
• Control group (cgroup): the kernel feature systemd uses to track every process in a service.

Go deeper

• systemctl cat ssh shows a real, packaged unit file. Compare it with yours.
• journalctl -u books-api -o json-pretty -n 1 shows everything the journal stores about one line, far more than it prints.
• Run vagrant halt when you're done for the day, and vagrant up next time. Your server and the service come back as you left them.
• Lesson 6, Processes and signals, for what systemd is doing for you underneath.