Resources Linux Linux for the cloud

Linux for the cloud · Lesson 10 of 15 · about 35 minutes

Disks, storage and memory

What fills up as a server runs, and what happens when it does. Read df and lsblk, find what's using the space with du and find, add a disk and mount it at every boot, then fill it completely: the errors a full disk causes, the deleted file that still takes up space, and the safe way to empty a log. Then memory: what free really says, and the kernel's out-of-memory killer, caught in the act.

By the end you'll know

  • How to see how full every disk is, and what each one is for
  • How to find what's using the space, from folders down to single files
  • How to add a disk, put a filesystem on it, and mount it at every boot
  • What a full disk breaks, and how to free space safely, even from a file that's in use
  • How to read memory with free, and what the out-of-memory killer does

The problem: things fill up

Lesson 3 found a debug.log several times bigger than every other log put together, and the practice files' to-do list said: "Rotate the logs before the disk fills." Logs, caches, uploads and databases only grow. When a disk fills completely, things break in odd ways: services crash or refuse to start, databases stop accepting writes, and even logging in can fail, because the system can't write the record of you logging in. Memory runs out too, differently, and Linux has a blunt way of dealing with that.

This lesson looks at both, on the server. On your own computer, from your practice folder:

BASH
cd linux-practice
vagrant ssh

How full is everything?

BASH
df -h
OUTPUT
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              196M  1.2M  195M   1% /run
efivarfs                           256K   34K  223K  14% /sys/firmware/efi/efivars
/dev/mapper/ubuntu--vg-ubuntu--lv   30G  6.0G   23G  22% /
tmpfs                              979M  1.1M  978M   1% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/sda2                          2.0G  102M  1.7G   6% /boot
/dev/sda1                          1.1G  6.4M  1.1G   1% /boot/efi
tmpfs                              196M   12K  196M   1% /run/user/1000

df (disk free) lists every filesystem, the structure that organises files on a disk, and where it's mounted: the folder where its files appear. -h makes the sizes human-readable. The line that matters most is /, everything not on another filesystem: 30 GB, 22% used. /boot holds the kernel, the one Lesson 1 found missing in containers. The tmpfs lines aren't disks at all: they live in memory, and empty at every reboot. That's why /run is a good place for temporary files like the lock Lesson 8 mentioned, and a bad place for anything you need to keep.

Which disks are there? lsblk (list block devices) shows the hardware side:

BASH
lsblk
OUTPUT
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda                         8:0    0   64G  0 disk 
├─sda1                      8:1    0    1G  0 part /boot/efi
├─sda2                      8:2    0    2G  0 part /boot
└─sda3                      8:3    0 60.9G  0 part 
  └─ubuntu--vg-ubuntu--lv 252:0    0 30.5G  0 lvm  /
sr0                        11:0    1 1024M  0 rom

One disk, sda, 64 GB, cut into three partitions. The third, sda3, isn't mounted directly: it's managed by LVM (Logical Volume Manager), which lets you carve space out of partitions and grow it later. Here only 30.5 GB of its 60.9 GB is used for /, leaving room to grow without adding a disk. sr0 is a virtual CD drive. Disks on Linux are named by type: sda, sdb, and so on for most disks, nvme0n1 for the fast NVMe kind on many cloud servers.

What's using the space?

du (disk usage) adds up what a folder holds. -s gives one total per folder, and sort -h (human-readable sort) orders the results, biggest last:

BASH
sudo du -sh /var/* 2>/dev/null | sort -h | tail -5
journalctl --disk-usage
OUTPUT
52K	/var/tmp
676K	/var/backups
55M	/var/log
144M	/var/cache
299M	/var/lib
Archived and active journals take up 29.6M in the file system.

/var/lib is the biggest: that's where PostgreSQL and apt keep their data (Lesson 2's "data that changes"). /var/log is 55 MB, and the journal accounts for about half of it. On a busy server, /var/log is the usual suspect. Run du again on whichever folder is biggest, sudo du -sh /var/lib/* | sort -h | tail, and keep going down until you find it.

Or go straight to big files. Lesson 3's find -size, across the whole disk:

BASH
sudo find / -xdev -type f -size +50M -exec ls -lh {} \; 2>/dev/null | head -5
OUTPUT
-rw-r--r-- 1 root root 59M Oct 30 00:13 /var/cache/apt/srcpkgcache.bin
-rw-r--r-- 1 root root 59M Oct 30 00:20 /var/cache/apt/pkgcache.bin
-rw-r--r-- 1 root root 69M Apr 25  2024 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_noble_universe_binary-arm64_Packages
-rw------- 1 root root 3.9G Oct 23  2025 /swap.img
-rw-r--r-- 1 root root 115M Apr 14  2024 /usr/lib/aarch64-linux-gnu/libLLVM-17.so.1

-xdev stays on one filesystem, so find doesn't wander into /proc or other disks. The biggest file by far is /swap.img, 3.9 GB, which is memory spilling onto disk (more at the end). The apt caches and lists come from Lesson 8; sudo apt clean deletes the downloaded packages if space is tight. The 115 MB library came with PostgreSQL. Nothing here is a mistake, and that's normal: most disk space is used by something on purpose. The job is finding what grew unexpectedly.

Add a disk

The bookshop needs somewhere for its own data, separate from the system: if it fills up, the server itself keeps working. On a cloud server you'd attach a new volume (a virtual disk), and it would appear in lsblk as sdb or nvme1n1. Here, a file can play the part of a disk, called a loop device, which lets you try every step safely:

BASH
sudo fallocate -l 100M /var/tmp/books-data.img
sudo mkfs.ext4 -q /var/tmp/books-data.img
sudo mkdir -p /srv/books-data
sudo mount -o loop /var/tmp/books-data.img /srv/books-data
lsblk | grep loop
df -h /srv/books-data
findmnt /srv/books-data
sudo chown bookshop:bookshop /srv/books-data
OUTPUT
loop0                       7:0    0  100M  0 loop /srv/books-data
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       90M   24K   83M   1% /srv/books-data
TARGET          SOURCE     FSTYPE OPTIONS
/srv/books-data /dev/loop0 ext4   rw,relatime

The same three steps as a real disk:

  1. Make a filesystem with mkfs.ext4. ext4 is Linux's standard filesystem. On a real disk you'd name the device, sudo mkfs.ext4 /dev/sdb, and it erases everything on it: check the name with lsblk twice.
  2. Make an empty folder to mount it on, the mount point. /srv is Lesson 2's place for data a server serves.
  3. Mount it with mount. -o loop is only for files pretending to be disks; a real disk is just sudo mount /dev/sdb /srv/books-data.

lsblk now shows loop0 mounted at /srv/books-data. Notice df says 90M, not 100M, with 83M available: the filesystem itself takes some space, and ext4 keeps 5% back for root, so the system can still write when users have filled the disk. chown hands it to the app's user, as in Lesson 5.

Mount it at every boot

mount lasts until the next reboot. Mounts that should always be there are listed in /etc/fstab (filesystem table), one per line:

BASH
echo "/var/tmp/books-data.img /srv/books-data ext4 loop,nofail 0 2" | sudo tee -a /etc/fstab
sudo findmnt --verify
OUTPUT
/var/tmp/books-data.img /srv/books-data ext4 loop,nofail 0 2
none
   [W] non-bind mount source /swap.img is a directory or regular file
/srv/books-data
   [W] non-bind mount source /var/tmp/books-data.img is a directory or regular file
   [W] your fstab has been modified, but systemd still uses the old version;
       use 'systemctl daemon-reload' to reload

0 parse errors, 0 errors, 3 warnings
FieldHereMeans
What/var/tmp/books-data.imgThe disk, or here the file. Real disks are usually named by their UUID=..., from sudo blkid, which never changes, unlike sdb
Where/srv/books-dataThe mount point
Typeext4The filesystem
Optionsloop,nofailHow to mount it. nofail means: if it's missing, boot anyway
Dump, check order0 2Backups (unused, always 0), and the order to check it at boot: 1 for /, 2 for everything else

findmnt --verify checks the file before a reboot does. 0 errors is what matters. The first two warnings just say the source is a file, not a disk, which is true for both the loop file and the swap file. The third is Lesson 7's rule again: systemd reads /etc/fstab into its own mount units, so after changing it, run sudo systemctl daemon-reload. Never skip this check. A broken /etc/fstab line without nofail can stop a server booting at all, and on a cloud server, you can't just walk up to it and fix it.

Test it without rebooting: unmount the disk, then mount everything /etc/fstab lists:

BASH
sudo umount /srv/books-data
findmnt /srv/books-data
sudo mount -a
findmnt /srv/books-data
OUTPUT
mount: (hint) your fstab has been modified, but systemd still uses
       the old version; use 'systemctl daemon-reload' to reload.
TARGET          SOURCE     FSTYPE OPTIONS
/srv/books-data /dev/loop0 ext4   rw,relatime

After umount (no "n"), findmnt printed nothing: not mounted. mount -a mounted it again from /etc/fstab, with the same reminder. Run sudo systemctl daemon-reload now, and after your next vagrant reload it'll come back by itself.

Break it on purpose: a full disk

Now let the app's debug logging do what the practice files feared. yes prints the same line forever; here, into a log on the small disk. The mountpoint check runs first, and && (Lesson 4's "only if that worked") means nothing is written unless the small disk really is mounted, so a mistake can't fill your real one:

BASH
mountpoint /srv/books-data && sudo -u bookshop bash -c 'yes "2026-10-30 01:00:31.281 DEBUG request id=88080b path=/api/books took=52ms" > /srv/books-data/debug.log'
df -h /srv/books-data
ls -lh /srv/books-data
echo "a note" | sudo -u bookshop tee /srv/books-data/note.txt
OUTPUT
/srv/books-data is a mountpoint
yes: standard output: No space left on device
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       90M   83M     0 100% /srv/books-data
total 83M
-rw-rw-r-- 1 bookshop bookshop 83M Oct 30 13:46 debug.log
drwx------ 2 root     root     16K Oct 30 13:46 lost+found
a note
tee: /srv/books-data/note.txt: No space left on device

In about a second, an 83M log, and No space left on device: the error every program gives when a disk is full. df says 100%, 0 available. And the note: tee printed "a note" to the screen, as it always does, but couldn't save it. The file note.txt was created, empty, and then there was no room for its contents. That's typical of a full disk: things half-work, which is more confusing than a clean failure. (lost+found is where ext4 puts pieces of files it recovers after a crash.)

The deleted file that still fills the disk

The obvious fix is to delete the log. But real logs are open: the program writing them doesn't stop. Pretend a program is still reading the log, with tail -f (Lesson 3), then delete it:

BASH
sudo -u bookshop bash -c 'tail -f /srv/books-data/debug.log > /dev/null &'
sudo rm /srv/books-data/debug.log
ls -lh /srv/books-data
df -h /srv/books-data
OUTPUT
total 16K
drwx------ 2 root     root     16K Oct 30 13:46 lost+found
-rw-rw-r-- 1 bookshop bookshop   0 Oct 30 13:46 note.txt
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       90M   83M     0 100% /srv/books-data

ls says it's gone. df says the disk is still full. On Linux, rm removes a file's name, not its contents. The contents are only freed when nothing is using them any more, and tail still has the file open. Find which program, through /proc from Lesson 6, where every process's open files are listed under fd (file descriptors):

BASH
sudo find /proc/*/fd -lname '*deleted*' -printf '%p -> %l\n' 2>/dev/null
OUTPUT
/proc/10336/fd/2 -> /dev/pts/1 (deleted)
/proc/10336/fd/3 -> /srv/books-data/debug.log (deleted)

Process 10336, file descriptor 3: debug.log (deleted). (The other line is the terminal tail was started from, closed since; harmless.) lsof +L1 (list open files with no name left) answers the same question, more readably. Stop the program, and the space comes back:

BASH
sudo pkill -u bookshop -x tail
sleep 1
df -h /srv/books-data
OUTPUT
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       90M   24K   83M   1% /srv/books-data

From 100% to 1%, the instant the last program let go. On a real server, that program is usually the service writing the log, so the fix is sudo systemctl restart on it. Better still is not to get there at all.

Empty a log instead of deleting it. truncate -s 0 cuts a file to 0 bytes in place: same name, same file, so the program writing it carries on, and the space is free at once:

BASH
sudo -u bookshop bash -c 'yes "DEBUG filler" | head -c 50M > /srv/books-data/debug.log'
df -h /srv/books-data
sudo truncate -s 0 /srv/books-data/debug.log
ls -lh /srv/books-data/debug.log
df -h /srv/books-data
OUTPUT
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       90M   51M   33M  61% /srv/books-data
-rw-rw-r-- 1 bookshop bookshop 0 Oct 30 13:47 /srv/books-data/debug.log
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       90M   24K   83M   1% /srv/books-data

Try all three fixes, in any order:

Try it: free up a full disk

ls -lhdebug.log, 83M
df -h83M used, 100%
the programrunning, writing debug.log

The disk is full: a program is writing debug.log, and it has filled all 83M.

One more way to be full. Every file also uses an inode, a slot in the filesystem's index of files, and there's a fixed number of them:

BASH
df -i /srv/books-data
OUTPUT
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop0      25600    13 25587    1% /srv/books-data

25,600 slots, 13 used. Millions of tiny files, like an app's session files or a mail queue, can use every inode while df -h still shows free space. If you get No space left on device and df -h looks fine, run df -i.

Memory

BASH
free -h
OUTPUT
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       336Mi       488Mi        27Mi       1.2Gi       1.6Gi
Swap:          3.8Gi          0B       3.8Gi

The server has 1.9 GB of memory (the 2048 MB in the Vagrantfile, minus what the kernel keeps for itself). Only 488 MB is free, which looks alarming and isn't:

  • buff/cache, 1.2 GB, is memory Linux uses to keep recently read files at hand, so reading them again is fast. It's handed back the moment a program needs it. Unused memory is wasted memory, so Linux fills it.
  • available, 1.6 GB, is the number to watch: how much programs could still get, cache included. When that runs low, the server is short of memory.
  • Swap is disk space used as overflow memory, here the 3.9 GB /swap.img. It's far slower than memory. 0 used is healthy; a server constantly using swap is short of memory, and slow because of it.

free reads all this from /proc/meminfo, and ps from Lesson 6 says who's using it:

BASH
grep -E '^(MemTotal|MemFree|MemAvailable|Cached)' /proc/meminfo
ps aux --sort=-%mem | head -4
swapon --show
OUTPUT
MemTotal:        2003572 kB
MemFree:          499772 kB
MemAvailable:    1659252 kB
Cached:          1024636 kB
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
postgres    5562  0.0  1.5 224252 30080 ?        Ss   11:44   0:00 /usr/lib/postgresql/16/bin/postgres -D /var/lib/postgresql/16/main -c config_file=/etc/postgresql/16/main/postgresql.conf
root         472  0.0  1.3 290464 26368 ?        SLsl 11:20   0:01 /sbin/multipathd -d -s
postgres    5563  0.0  1.1 224540 22928 ?        Ss   11:44   0:00 postgres: 16/main: checkpointer 
NAME      TYPE SIZE USED PRIO
/swap.img file 3.8G   0B   -2

PostgreSQL is the biggest user, at 1.5%, about 30 MB (RSS, in kilobytes). This server is nowhere near its limit.

Break it on purpose: out of memory

When memory really runs out, the kernel has no good option. It picks a process and kills it with KILL, signal 9: the OOM killer (out of memory). You can watch it safely. systemd-run starts a one-off service, and MemoryMax=50M gives it a 50 MB limit, the same control-group feature from Lesson 7. Then ask Python for 200 MB:

BASH
sudo systemd-run --unit=memtest -p MemoryMax=50M -p MemorySwapMax=0 python3 -c 'x = b"a" * (200 * 1024 * 1024)'
sleep 3
journalctl -u memtest --no-pager -n 6
OUTPUT
Running as unit: memtest.service; invocation ID: e173dbd9df874ce2a6a3bede7a59b296
Oct 30 13:47:24 bookshop systemd[1]: Started memtest.service - /usr/bin/python3 -c "x = b\"a\" * (200 * 1024 * 1024)".
Oct 30 13:47:24 bookshop systemd[1]: memtest.service: A process of this unit has been killed by the OOM killer.
Oct 30 13:47:24 bookshop systemd[1]: memtest.service: Main process exited, code=killed, status=9/KILL
Oct 30 13:47:24 bookshop systemd[1]: memtest.service: Failed with result 'oom-kill'.

Killed by the OOM killer, status=9/KILL, in the same second it started. -p MemorySwapMax=0 stops it using swap instead. The kernel keeps its own record, which journalctl -k (kernel messages) shows:

BASH
sudo journalctl -k --no-pager -n 4 | grep -i -E 'oom|killed'
OUTPUT
Oct 30 13:47:24 bookshop kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Oct 30 13:47:24 bookshop kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/system.slice/memtest.service,task_memcg=/system.slice/memtest.service,task=python3,pid=10400,uid=0
Oct 30 13:47:24 bookshop kernel: Memory cgroup out of memory: Killed process 10400 (python3) total-vm:222072kB, anon-rss:50944kB, file-rss:6016kB, shmem-rss:0kB, UID:0 pgtables:172kB oom_score_adj:0

Memory cgroup out of memory: Killed process 10400 (python3), with anon-rss:50944kB: it reached its 50 MB and no further. This is the 137 from Lesson 6 (128 + 9) and from the Docker track: a container killed for using too much memory gets exactly this, because a container's memory limit is the same control-group feature. When a service keeps dying with oom-kill or 137, give it more memory, or find out why it wants so much.

On a real server

  • Rotate logs. logrotate runs daily on Ubuntu, and renames, compresses and eventually deletes old logs. cat /etc/logrotate.d/nginx shows nginx's rules: that's where access.log.1, from Lesson 3's practice files, comes from. Logs going to the journal are limited by SystemMaxUse= in /etc/systemd/journald.conf.
  • Alert before it's full. Monitoring should warn at about 80% disk use, while there's still time to act. At 100%, you're fixing it in a hurry.
  • Grow, don't just clean. Cloud volumes can be made bigger while the server runs. Then the partition, the LVM volume and the filesystem are each grown to match: growpart, lvextend, resize2fs. Look up your provider's steps before you need them.
  • df and du disagree? df counts deleted files that are still open; du only sees files with names. A big difference means sudo lsof +L1.
  • Mount data on its own disk, like /srv/books-data, or databases under /var/lib/postgresql. A runaway log then can't take the whole server down with it.

Spot the problem

Work out what's wrong before you read the explanation.

Deleted, and still full. At 3 in the morning a disk alert fires: /var/log is at 100%. Someone runs sudo rm /var/log/books-api/debug.log, a 20 GB file. ls shows it's gone, but df still says 100%, and the alert keeps firing. What's going on, and what's the fix? The app still has the file open, so its 20 GB can't be freed. sudo lsof +L1 shows it; sudo systemctl restart books-api makes the app let go, and the space is freed at once. Next time, sudo truncate -s 0 the file instead of deleting it, and set up rotation so it never gets to 20 GB again.

Keep this

ToRun
How full is every filesystemdf -h
Out of inodes?df -i
Disks, partitions and mountslsblk
Biggest folders, biggest lastsudo du -sh /var/* | sort -h | tail
Big files, on one filesystemsudo find / -xdev -type f -size +100M -exec ls -lh {} ;
How much the journal usesjournalctl --disk-usage
Make a filesystem (erases the disk)sudo mkfs.ext4 /dev/sdb
Mount, and unmountsudo mount /dev/sdb /srv/data, sudo umount /srv/data
Where something is mountedfindmnt /srv/data
A disk's unchanging ID, for fstabsudo blkid
Check fstab before a reboot doessudo findmnt --verify, then sudo systemctl daemon-reload
Mount everything in fstabsudo mount -a
Empty a log that's in usesudo truncate -s 0 FILE
Deleted files still taking spacesudo lsof +L1
Memory: read the available columnfree -h
Biggest users of memoryps aux --sort=-%mem | head
Is there swap?swapon --show
Was anything killed for memory?sudo journalctl -k | grep -i oom
Clear downloaded packagessudo apt clean

Where you are now

You know what fills a server and how to find it, how to add a disk and keep it mounted, what a full disk and a memory shortage actually do, and how to free space without being fooled by a deleted file. Lesson 11 is SSH and keys: how vagrant ssh really works, making your own key, logging in without Vagrant's help, and locking SSH down.

Check yourself

  1. free -h shows 488Mi free and 1.6Gi available. Is the server short of memory?

  2. You deleted a 20 GB log, but df still shows the disk full. Why?

  3. What's the safe way to empty a log that a running app is writing?

  4. Why add nofail to a line in /etc/fstab?

  5. A service keeps failing with result 'oom-kill'. What happened?

  6. df -h shows space free, but writing still fails with No space left on device. What do you check?

Key terms

  • Filesystem: the structure that organises files on a disk. ext4 is Linux's standard one.
  • Mount, and mount point: attaching a filesystem so its files appear in a folder, and that folder.
  • Partition and LVM: a fixed slice of a disk, and a layer on top that lets space be carved out and grown later.
  • Loop device: a file used as if it were a disk.
  • /etc/fstab: the list of filesystems to mount at every boot.
  • Inode: a slot in a filesystem's index of files. Every file uses one.
  • Cache and swap: spare memory used to keep files at hand, and disk space used as slow overflow memory.
  • OOM killer: the kernel's way of freeing memory when there's none left: it kills a process.

Go deeper

  • sudo du -sh /var/lib/* | sort -h | tail goes one level deeper. What's the biggest thing in /var/lib?
  • cat /etc/logrotate.d/nginx, then ls -l /var/log/nginx after tomorrow's rotation.
  • sudo lsof +L1 right now: are any of this server's services holding deleted files?
  • Lesson 6, Processes and signals, for exit code 137 and KILL.

From www.kwamib.dev/resources/linux/disks-storage-and-memory