Auditing a Linux server

Hardening is a moment; auditing is ongoing

Lesson 3 left the server in good shape. But servers drift: someone opens a port to debug something and forgets it, a package installs a service, an attacker adds an account. Auditing answers three questions, over and over:

1. How does this server measure up? A checklist, scored: Lynis.
2. Who changed what, and when? A record kept as it happens: auditd.
3. What's different from how it should be? A known-good description to compare against: a baseline.

Each answers something the others can't, which this lesson shows by trying all three on the same change.

1. The tools

sudo apt-get install -y lynis auditd
lynis show version; auditctl -v; systemctl is-active auditd

lynis 3.0.9
auditctl version 3.1.2
active

• Lynis is a free, open-source auditing tool by CISOfy: one shell script that runs a few hundred checks.
• auditd is the Linux audit daemon. The kernel itself reports events (a file opened for writing, a program started) to it, so a program can't avoid being recorded by not cooperating.

2. Lynis: a score against a checklist

sudo lynis audit system --quick --no-colors > /var/log/lynis-run-1.txt

• audit system runs every check that applies to this server.
• --quick doesn't stop to wait for you between sections.
• The details go to /var/log/lynis-report.dat, one finding per line, which is what the summary below reads.

It took 12 seconds:

hardening index: 67 / 100
tests run: 257
warnings: 1   suggestions: 45
warnings:
  PKGS-7388: Can't find any security repository in /etc/apt/sources.list or sources.list.d directory
suggestions (first 15):
  LYNIS: This release is more than 4 months old. Check the website or GitHub to see if there is an update available.
  DEB-0280: Install libpam-tmpdir to set $TMP and $TMPDIR for PAM sessions
  DEB-0810: Install apt-listbugs to display a list of critical bugs prior to each APT installation.
  DEB-0811: Install apt-listchanges to display any significant changes prior to any upgrade via APT.
  DEB-0880: Install fail2ban to automatically ban hosts that commit multiple authentication errors.
  BOOT-5122: Set a password on GRUB boot loader to prevent altering boot configuration (e.g. boot in single user mode without password)
  BOOT-5264: Consider hardening system services
  KRNL-5820: If not required, consider explicit disabling of core dump in /etc/security/limits.conf file
  AUTH-9230: Configure password hashing rounds in /etc/login.defs
  AUTH-9262: Install a PAM module for password strength testing like pam_cracklib or pam_passwdqc
  AUTH-9282: When possible set expire dates for all password protected accounts
  AUTH-9284: Look at the locked accounts and consider removing them
  AUTH-9286: Configure minimum password age in /etc/login.defs
  AUTH-9286: Configure maximum password age in /etc/login.defs
  AUTH-9328: Default umask in /etc/login.defs could be more strict like 027

How to read it:

• The hardening index is Lynis's own score, not a standard. It's useful for comparing the same server over time, and not much else.
• The one warning is wrong. Ubuntu 24.04 lists its package sources, the security one included, in a newer file format (ubuntu.sources) that this version of Lynis doesn't read. Every tool has false alarms like this; checking them is part of the job.
• Suggestions aren't orders. Some are worth doing (fail2ban, a stricter umask). Some don't fit (a boot loader password on a cloud VM no one can reach the console of). Some are already handled: AUTH-9284 points at deploy, locked on purpose in Lesson 3. Write down why you skip one, so the next audit doesn't ask again.
• The first suggestion is about Lynis itself. Ubuntu's package is older than the version on GitHub, which has newer checks.

3. auditd: a record of who changed what

The rules, in /etc/audit/rules.d/50-lesson.rules:

-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-w /etc/ssh/sshd_config -p wa -k sshd
-w /etc/ssh/sshd_config.d/ -p wa -k sshd
-w /etc/crontab -p wa -k cron
-w /etc/cron.d/ -p wa -k cron
-w /usr/local/bin/ -p wa -k local_bin
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=4294967295 -k admin_cmds

• -w PATH watches a file, or everything in a folder.
• -p wa records writes and attribute changes (permissions, owner, dates), not reads.
• -k identity tags each record with a key, so you can search for it later.
• The last rule records every program started (execve) as root (euid=0) by someone who logged in as a person: a login user ID (auid) of 1000 or more, and set (4294967295 means "never logged in", which covers services started at boot).

sudo augenrules --load
sudo auditctl -l

• augenrules --load combines the files in rules.d/ and loads them.
• auditctl -l lists the rules the kernel is using now.

-w /etc/passwd -p wa -k identity
...
-w /usr/local/bin -p wa -k local_bin
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=-1 -F key=admin_cmds

All eleven loaded.

4. A baseline, kept off the server

A baseline is a description of the server as it should be. Here it's a short script, snapshot, that prints what matters most as plain text: accounts that can log in, who has user ID 0, the sudo group, a fingerprint (part of the SHA-256 hash) of each sudoers and cron file, the SSH settings in force, every authorized SSH key, what's in /usr/local, listening ports, firewall rules and enabled services.

multipass exec sec-server -- sudo snapshot > ~/Apps/security-lesson04/baseline.txt

The output goes to the Mac, not a file on the server. Anyone who can change the server can change a baseline stored on it.

      90 /Users/mayowababatola/Apps/security-lesson04/baseline.txt

## accounts that can log in (name:uid)
root:0
sync:4
ubuntu:1001
ops:1002
## accounts with uid 0
root
## sudo group
sudo:x:27:ubuntu,ops
## sudoers files (first 16 characters of each file's sha256)
5bac27ce5ff1a78a  /etc/sudoers
3296cc0b35cbc477  /etc/sudoers.d/90-cloud-init-users

The script also left an empty file, /root/.baseline-marker, to mark the moment.

5. Someone else's turn

Then, at 15:10, the script logged in as ops, with its key and sudo password, and changed the server. The output hid what it did. In a real incident, this is someone who stole a laptop with the key on it, or a colleague doing something "just for now".

Before reading on: what would you check first?

6. Hunting by timestamps

Every file has a modification time (mtime): when its contents last changed. That's what ls -l shows. Find everything newer than the marker:

sudo find /etc /usr/local /home /var/spool/cron -type f -newer /root/.baseline-marker

/etc/group
/etc/gshadow
/etc/passwd
/etc/shadow
/etc/subgid
/etc/subuid
/home/ops/.sudo_as_admin_successful
/home/svc-backup/.ssh/authorized_keys

A new account, svc-backup, with an SSH key. That looks like everything, and it isn't. Files also have a change time (ctime): when anything about the file last changed, contents or dates or permissions. Anyone can set a file's mtime to whatever they like with touch, but doing so sets its ctime to now, and nothing short of changing the system clock can set it back.

sudo find /etc /usr/local /home /var/spool/cron -type f -cnewer /root/.baseline-marker

/etc/cron.d/backup-sync
/etc/group
...
/etc/ssh/sshd_config.d/10-hardening.conf
...
/etc/sudoers.d/99-svc-backup
...
/usr/local/bin/backup-sync

Four files the first search missed. Here they are with both times (the stat command shows them):

/etc/cron.d/backup-sync   mtime 2024-03-30 20:06:27   ctime 2026-11-07 15:10:20
/etc/ssh/sshd_config.d/10-hardening.conf   mtime 2026-11-07 09:06:12   ctime 2026-11-07 15:10:20
/etc/sudoers.d/99-svc-backup   mtime 2024-01-29 12:09:56   ctime 2026-11-07 15:10:20
/usr/local/bin/backup-sync   mtime 2024-03-30 20:06:27   ctime 2026-11-07 15:10:20

Their modification dates were set back, to match nearby files: ls -l /etc/sudoers.d/ would show 99-svc-backup as almost three years old. A file whose mtime is far older than its ctime is worth a look all by itself. (The other files in this list, /etc/passwd- and the like, are the backups useradd makes; it keeps their old dates on purpose. And the dotfiles in svc-backup's home were copied from /etc/skel with their original dates.)

7. The baseline, compared

multipass exec sec-server -- sudo snapshot > ~/Apps/security-lesson04/now.txt
diff -u ~/Apps/security-lesson04/baseline.txt ~/Apps/security-lesson04/now.txt

• diff -u shows lines removed (-) and added (+), with a few unchanged lines around each for context.

@@ -3,6 +3,7 @@
 sync:4
 ubuntu:1001
 ops:1002
+svc-backup:1003
 ## accounts with uid 0
 root
 ## sudo group
@@ -10,9 +11,11 @@
 ## sudoers files (first 16 characters of each file's sha256)
 5bac27ce5ff1a78a  /etc/sudoers
 3296cc0b35cbc477  /etc/sudoers.d/90-cloud-init-users
+431656045cb9b1fb  /etc/sudoers.d/99-svc-backup
 b428c9b673c3c806  /etc/sudoers.d/README
 ## ssh settings in force
 allowusers ops
+allowusers svc-backup
 allowusers ubuntu
 passwordauthentication no
 permitrootlogin no
@@ -20,13 +23,16 @@
 ## authorized ssh keys
 /root/.ssh/authorized_keys: no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo ubuntu@localhost
 /home/ops/.ssh/authorized_keys: ssh-ed25519 (mayowababatola@KwamiB)
+/home/svc-backup/.ssh/authorized_keys: ssh-ed25519 backup@build-07
 /home/ubuntu/.ssh/authorized_keys: ssh-rsa ubuntu@localhost
 ## cron files
 ffb48ad57868ed63  /etc/crontab
+22b01faa694a06ff  /etc/cron.d/backup-sync
 06b70de96f9f3010  /etc/cron.d/e2scrub_all
 47a67deba0d029d4  /etc/cron.d/sysstat
 ## /usr/local/bin and /usr/local/sbin
 /usr/local/bin:
+backup-sync

The whole story, in one diff:

1. A new account, svc-backup.
2. A sudoers file for it. (Inside: svc-backup ALL=(ALL) NOPASSWD:ALL, root with no password.)
3. svc-backup added to SSH's AllowUsers, so it can log in.
4. An SSH key from someone calling themselves backup@build-07.
5. A cron job, backup-sync.
6. The script it runs, every 15 minutes, as root. (Inside: tar czf /var/tmp/.cache.tgz /etc/shadow /home, a hidden archive of every password hash and home folder, ready to collect.)

The names are chosen to look routine. The baseline doesn't care about names or dates: it only knows the server is different.

8. The audit log: who, when, how

ausearch searches the audit log by key. --format text turns each record into a sentence:

sudo ausearch --input-logs -m SYSCALL -k sudoers --format text

• --input-logs reads the log files. Without it, when ausearch isn't attached to a terminal (in a script, or through multipass exec), it waits for records on its input and appears to hang. That happened while writing this lesson.
• -m SYSCALL shows only system call records: the actions themselves.

-- key: identity
At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /etc/passwd using /usr/sbin/useradd
At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /etc/group using /usr/sbin/useradd
At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /etc/shadow using /usr/sbin/useradd
-- key: sudoers
At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /etc/sudoers.d/99-svc-backup using bash
At 15:10:20 11/07/26 ops, acting as root, successfully changed-file-permissions-of /etc/sudoers.d/99-svc-backup to 0440 using /usr/bin/chmod
At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /etc/sudoers.d/99-svc-backup using /usr/bin/touch
-- key: sshd
At 15:10:20 11/07/26 ops, acting as root, successfully renamed /etc/ssh/sshd_config.d/sedVdQZGo to /etc/ssh/sshd_config.d/10-hardening.conf using /usr/bin/sed
At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /etc/ssh/sshd_config.d/10-hardening.conf using /usr/bin/touch
-- key: cron
At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /etc/cron.d/backup-sync using bash
At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /etc/cron.d/backup-sync using /usr/bin/touch
-- key: local_bin
At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /usr/local/bin/backup-sync using bash
At 15:10:20 11/07/26 ops, acting as root, successfully changed-file-permissions-of /usr/local/bin/backup-sync to 0755 using /usr/bin/chmod
At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /usr/local/bin/backup-sync using /usr/bin/touch

(Extract. Each key also starts with lines from 15:10:16, ubuntu, acting as root, successfully add_rule: the rules being loaded, which is recorded too.)

Every change, to the second, with the program that made it. Even the touch that set the dates back is there: hiding the change left its own record. sed -i, it turns out, writes a temporary file (sedVdQZGo) and renames it over the original.

Why it says "ops", not "root"

ops, acting as root. Every command ran as root, through sudo, and the log still names ops. Here are the fields behind that sentence, from one raw record (this one is the rules being loaded):

auid=ubuntu uid=root euid=root comm=auditctl exe=/usr/sbin/auditctl

• uid and euid: who the program ran as. Root.
• auid, the login user ID: who logged in to start this session. It's set once, at login, and kept by everything started from that session, sudo and su included. The person behind ops can become root; they can't stop being ops in the log.

And the last rule, every program run as root by whoever logged in as ops:

sudo ausearch --input-logs -k admin_cmds -ul $(id -u ops) -i | grep '^type=EXECVE'

• -ul matches the login user ID.
• -i turns numbers into names.
• EXECVE records hold each program's arguments.

sh -c -- /usr/bin/env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin run-parts --lsbsysinit /etc/update-motd.d > /run/motd.dynamic.new
/usr/bin/env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin run-parts --lsbsysinit /etc/update-motd.d
run-parts --lsbsysinit /etc/update-motd.d
/bin/sh /etc/update-motd.d/00-header
uname -o
uname -r
...
/bin/sh /etc/update-motd.d/98-fsck-at-reboot

The first 40 lines were all noise. Every SSH login runs Ubuntu's login-message scripts (update-motd.d) as root, after the login user ID is set, so each login records about 40 programs before the person types anything. The commands that matter were further down. Recording every program run as root is a common rule, and this is its cost: it's thorough, and most of what it records isn't interesting. Searching by file key, as above, finds the important part much faster.

9. Does what they left work?

From the Mac, with the other person's key:

ssh -i ~/.ssh/sec-lesson-other-ed25519 svc-backup@192.168.252.3 'whoami; sudo -n id'

svc-backup
uid=0(root) gid=0(root) groups=0(root)

Root, without a password. On a server hardened in Lesson 3 (keys only, no root login, named users only, a firewall), because someone with legitimate access added a way back in. Hardening limits who gets in; it can't stop an insider, or someone with an insider's key.

10. Lynis again

Same command, run while the backdoor was there:

run 1: index 67, warnings 1, suggestions 45
run 2: index 68, warnings 1, suggestions 44

Findings in run 2 that weren't in run 1 (> new, < gone):
< suggestion[]=ACCT-9630|Audit daemon is enabled with an empty ruleset. Disable the daemon or define rules

The score went up. One suggestion went away: auditd now had rules, which this lesson added. Nothing about the new account, the sudoers file, the key or the cron job.

11. Undo it, and check

sudo userdel -r svc-backup
sudo rm -fv /etc/sudoers.d/99-svc-backup /etc/cron.d/backup-sync /usr/local/bin/backup-sync /var/tmp/.cache.tgz
sudo sed -i 's/^AllowUsers ops ubuntu svc-backup$/AllowUsers ops ubuntu/' /etc/ssh/sshd_config.d/10-hardening.conf
sudo visudo -c -q && sudo sshd -t && sudo systemctl reload ssh && echo removed, settings valid, ssh reloaded

removed '/etc/sudoers.d/99-svc-backup'
removed '/etc/cron.d/backup-sync'
removed '/usr/local/bin/backup-sync'
removed, settings valid, ssh reloaded

• userdel -r removes the account and its home folder, key included.
• No /var/tmp/.cache.tgz to remove: the cron job would first have run at 15:15, and this was 15:14.
• sed takes svc-backup back out of AllowUsers, and visudo -c and sshd -t check the settings before SSH reloads.

$ ssh -i ~/.ssh/sec-lesson-other-ed25519 svc-backup@192.168.252.3 true
svc-backup@192.168.252.3: Permission denied (publickey).

$ diff -u baseline.txt after-cleanup.txt && echo 'no differences from the baseline'
no differences from the baseline

And the audit log recorded the clean-up, and who did it:

At 15:10:20 11/07/26 ops, acting as root, successfully opened-file /etc/sudoers.d/99-svc-backup using /usr/bin/touch
At 15:14:13 11/07/26 ubuntu, acting as root, successfully deleted /etc/sudoers.d/99-svc-backup using /usr/bin/rm

In a real incident, removing the backdoor is the last step, not the first: you'd keep copies of everything as evidence, and find out how the key was taken. That's Lesson 11.

What each one caught

The baseline says what. The audit log says who and when. Neither needs to know what an attacker would do: they only need to know what normal looks like.

Spot the problem

Work out what's wrong before you read the explanation.

The score that went up. A team reports a security score from Lynis to management every month. This month it went from 67 to 68: progress. Nobody looked further. What did the score miss? Everything that mattered. On this server, the month the score went up was the month someone added an account with root and no password, a stranger's SSH key and a cron job copying every password hash. Lynis checks settings against a list of good practice; it has no idea what accounts, keys or jobs should be there, so a well-built backdoor looks like normal administration. It even rose for a real improvement (audit rules added), which made it look like good news. Three fixes: report the score, but never as the only signal; compare the server against its own baseline, which caught all six changes; and read the audit log for changes to accounts, sudo and SSH, which named who made them. A rising score tells you settings improved. It doesn't tell you nobody got in.

Where you are now

You can score a server, record who changes it, and tell when it's different from how it should be, and you know which question each one answers. So far the server has been the whole story. The next lesson moves to the cluster, and to the thing every service needs and nobody should see: secrets.

Check yourself

Key terms

• Auditing: checking, again and again, that a system is as it should be, and recording how it changes.
• Lynis: a free tool that checks a Unix server against a few hundred good practices and gives it a score.
• Hardening index: Lynis's score out of 100. Useful for trends on one server, not as a standard.
• auditd: the Linux audit daemon, which records events the kernel reports.
• Audit rule: what to record: a watched file (-w) or a system call (-a), tagged with a key (-k).
• Login user ID (auid): the user who logged in to start a session, kept through sudo and su.
• Baseline: a description of a system as it should be, kept somewhere the system can't change it.
• mtime and ctime: when a file's contents last changed (can be set by anyone with write access), and when anything about it last changed (can't be set back).
• File integrity monitoring (FIM): comparing files against a baseline of their hashes, on a schedule.

Go deeper

• Lynis (opens in a new tab) and its source on GitHub (opens in a new tab), including what each test ID checks.
• Red Hat's guide to the Linux audit system (opens in a new tab), the clearest explanation of rules and records.
• AIDE (opens in a new tab), the file integrity checker included with most Linux distributions.
• Try: add -w /home/ -p wa -k home to the rules and run the script again. Does the audit log now catch the SSH key?
• Try: fix three Lynis suggestions you agree with, and write down why you're skipping three others. What happens to the score?