🦴

Something is eating the disk

The tomb was empty yesterday. Today it's 97% full. Nobody put anything in it.

A little storage volume called tomb keeps filling up by itself. df says it's almost full. du swears there's hardly anything in it. Both are telling the truth. Something hidden is in there, and something invisible is feeding on it. Then give the tomb more room while it's still in use, and make sure it comes back after every boot.

Case 03 of 10System managementTroubleshootingNeeds sudo

⚠️ This case needs sudo

Play it on a practice VM (a free Ubuntu VM, a spare Pi, a lab node), not a machine you care about.
Your real disks are never touched. The "disks" are three plain 256 MB files in /var/lib/breakroom-03, turned into loop devices. Linux can't tell the difference.
It creates an LVM volume group crypt with a volume tomb, one line in /etc/fstab (with nofail, so boot never waits on it), and one service. --clean removes all of it.
Needs lvm2 and lsof (sudo apt install -y lvm2 lsof). Loop devices vanish on reboot: after one, just run the script again. And before --clean, cd out of the tomb, or it can't be unmounted.

Get the case

Rule 4: read it before you run it, less breakroom-mission-03.sh. The top of the file lists everything it creates and how --clean removes it.

curl -O https://arunnetworkingpro.com/labs/breakroom-mission-03.sh
sudo bash breakroom-mission-03.sh

At any point, ask the house how many ghosts are left:

sudo bash breakroom-mission-03.sh --check

Part A: what's eating it?

Two witnesses, two stories

df -h /mnt/breakroom-03/tomb
sudo du -sh /mnt/breakroom-03/tomb

How full does each one say it is? Write both numbers down: the gap between them is the whole mystery.

Hint

df asks the filesystem how many blocks are in use. du walks the folders and adds up the files it can see. When they disagree, something is using space without being a visible file.

The hidden bones

Find the single biggest file in the tomb, even if it's hiding, and get rid of it.

Hint

Files and folders starting with a dot are hidden from plain ls. sudo find /mnt/breakroom-03/tomb -size +100M finds big ones anyway. So does sudo du -ah /mnt/breakroom-03/tomb | sort -h | tail.

The invisible eater

df dropped, but it's still far fuller than du says. Something is holding on to space that has no name. Find the process, then the service behind it, and stop it for good.

Hint

A file that's deleted while a program still has it open keeps using space until the program lets go. sudo lsof -a +L1 /mnt/breakroom-03/tomb lists open files with zero links (deleted ones) on that filesystem only. (Without -a, lsof treats the two conditions as OR and lists every deleted file on the machine.) systemctl status <pid> names the service. disable --now stops it and keeps it from coming back at boot.

Part B: a bigger tomb

Add the spare disk

There's a third disk file nobody is using: /var/lib/breakroom-03/disk3.img. Its loop device already exists. Add it to the crypt volume group.

Hint

sudo losetup -j /var/lib/breakroom-03/disk3.img tells you which /dev/loopN it is. Then it's two LVM steps: make it a physical volume (pvcreate), then add it to the group (vgextend crypt ...). Check with sudo vgs and sudo pvs.

Grow it while it's open

Give tomb all the new free space, and grow the filesystem on it, without unmounting. df -h should show about 700 MB.

Hint

lvextend grows the volume. -l +100%FREE takes everything free, and -r resizes the filesystem in the same step (without it, the volume grows but df doesn't change).

Part C: a promise in fstab

The wrong address

Unmount the tomb and ask Linux to mount everything in /etc/fstab. It doesn't come back. Why?

sudo umount /mnt/breakroom-03/tomb
sudo mount -a
sudo findmnt --verify

Hint

findmnt --verify checks every fstab line. Compare the UUID in the breakroom-03 line with the real one from sudo blkid /dev/crypt/tomb. Can't unmount? Something still has a file open in there (see A3).

Keep the promise

Fix the fstab line, reload, mount everything again, and prove the tomb is back. Keep the # breakroom-03 at the end of the line, so --clean can find it later.

Hint

Edit with sudo nano /etc/fstab and fix only the UUID. Then sudo systemctl daemon-reload (systemd builds its mount units from fstab), sudo mount -a, and findmnt /mnt/breakroom-03/tomb.

✅ How you know you won

sudo bash breakroom-mission-03.sh --check says Verdict: 6/6. Nothing was eating the disk but Linux. Case closed.

Answer key (no peeking until you've tried)

A1: df says about 97% (321 MB used), du only about 171 MB. A2: /mnt/breakroom-03/tomb/.cellar/.bones (170 MB). A3: lsof -a +L1 shows python3 holding .feast.log (deleted); it belongs to breakroom-03-eater.service. sudo systemctl disable --now breakroom-03-eater. B1: sudo pvcreate /dev/loopN, sudo vgextend crypt /dev/loopN. B2: sudo lvextend -r -l +100%FREE crypt/tomb. C1: the fstab UUID is one character off; findmnt --verify says unreachable on boot required source. C2: put the real UUID from blkid in the line, daemon-reload, mount -a.

💥 Let it haunt you again

Run the script again. The haunting starts over, fresh:

sudo bash breakroom-mission-03.sh

Now do it without the hints. And when you're done for good: sudo bash breakroom-mission-03.sh --clean.

🧠 The ghost, explained

Verdict: not a ghost. The hidden bones were a dot-file, the invisible eater was a deleted file still held open, and the tomb that wouldn't come back was a typo in fstab.

df and du answer different questions. du adds up files it can see. df asks the filesystem what's in use. Deleted-but-open files are counted by df and invisible to du, which is why a log file deleted under a running service is a classic "disk full, but nothing's there" mystery. The fix is restarting (or stopping) the program, not deleting more files.

LVM is Lego for disks. Physical volumes (real disks or partitions) go into a volume group, one big pool. Logical volumes are cut from the pool and can grow while in use. pvs, vgs and lvs show each layer.

Growing is two jobs. The volume gets bigger, then the filesystem inside it has to be told. lvextend -r does both (it calls resize2fs or xfs_growfs for you). Shrinking is the dangerous direction: ext4 must be unmounted to shrink, and XFS can't shrink at all.

Use UUIDs in fstab, and check them. Device names like /dev/sdb can change between boots; UUIDs don't. findmnt --verify catches typos before a reboot does. And nofail means a missing disk won't drop the whole machine into emergency mode.

← Case 02 · All cases · Case 04 → · Stuck, or found a better way? Email me

🎉 Got it, thank you!

Your comment just landed in my inbox. I read every one, and I'll reply by email.

🤔 That didn't go through

Something in the form looked off. Check your name, email and comment and try again, or just email me.

🐢 Whoa, slow down

That's a lot of comments in a short time, so the box is taking a breather. Try again later, or email me.

😴 The comment box is napping

My server is taking a quick break, so your comment couldn't be sent. Sorry! Please email me instead.

💬 Leave a comment

Stuck, found a better way, or just built it and want to brag? Tell me. It comes straight to my inbox (nothing is posted publicly), and I'll reply by email.

Your email is only used to reply to you. Never shared, never added to any list.