ArunNetworkingPro

Let's break Linux 🔨 (and fix it)

Pick a patient. A ghost wrecks it and reboots it. You bring it back to life.

Every disaster here happens to real servers. Better to meet them here first, with a ghost doctor and a reload button.

⚠️ Do not try this at home. Or at work.

These breaks really stop a machine from booting. Don't do them to your laptop, your server, or your boss's server. Break this one: it's a real Debian computer I host on this site, and it runs inside your browser tab. Nothing you do reaches anything else. Reload the page and you get a brand-new patient.

🩺 How the ER works

  1. Pick a patient below. A healthy machine wakes up (about 44 MB, best on Wi-Fi).
  2. The ghost grabs the keyboard, breaks something and reboots. Watch the screen. Reboots are slow in a browser: give it a minute or two.
  3. Fix it. The patient's chart (further down) has hints, then the fix if you're stuck.
  4. Back at a normal prompt, type er-checkup. Really cured? You get the badge.

Phone? The keys under the screen are the ones phones don't have. GRUB menus wait only 5 seconds: tap an arrow key to freeze them.

One patient per visit. Reload for another.

Patient 1: GRUB ate its homework

What the patient says:

error: file '/boot/grub/i386-pc/normal.mod' not found. grub rescue>

Hint 1: what is this place?

GRUB is the bootloader: the tiny program that loads Linux. It keeps its brains in /boot/grub. That folder is gone, so GRUB is stuck in rescue mode. It only knows a few words: ls, set, insmod.

Hint 2: go looking

ls lists the disks. Then look inside the partition:

ls
ls (hd0,msdos1)/boot/

One folder there has a very suspicious name.

🩹 The fix (spoiler)

Point GRUB at its stolen brains, load the normal menu, boot:

set prefix=(hd0,msdos1)/boot/grub.eaten-by-ghost
set root=(hd0,msdos1)
insmod normal
normal
📌 Make it stick

It booted, but only because you held its hand. Put the folder back:

sudo mv /boot/grub.eaten-by-ghost /boot/grub
er-checkup

Why this happens to real servers:

Real life: a Windows install overwrote the boot sector, a disk got swapped, someone "cleaned up" /boot. The usual real fix is booting a live USB and running grub-install + update-grub.

Patient 2: one wrong letter

What the patient says:

Gave up waiting for root file system device... ALERT! LABEL=posessed does not exist. Dropping to a shell! (initramfs)

Hint 1: read the ALERT slowly

Really slowly. Letter by letter. How do you spell possessed?

Hint 2: where does that line come from?

GRUB hands the kernel a command line, and root= tells it which disk is the real system. You can change that line for one boot: reboot, and when the GRUB menu shows up press an arrow key to stop the 5-second countdown, then e to edit.

🩹 The fix (spoiler)

At (initramfs) type reboot -f. At the GRUB menu press e, fix the spelling on the linux line, then press Ctrl then x to boot. (Plan B: press c instead and type it all yourself:)

linux /boot/vmlinuz-6.1.0-53-686 root=LABEL=possessed console=ttyS0
initrd /boot/initrd.img-6.1.0-53-686
boot
📌 Make it stick

You fixed it for one boot. The typo still lives in /boot/grub/grub.cfg. Never hand-edit that file: regenerate it.

sudo update-grub
er-checkup

Why this happens to real servers:

Real life: someone hand-edited grub.cfg, or a disk got a new label or UUID. The settings live in /etc/default/grub; update-grub turns them into grub.cfg.

Patient 3: zombie in the initramfs

What the patient says:

BRAAAINS. A zombie crawled into the initramfs and is blocking the boot. (initramfs)

Hint 1: what is the initramfs?

A mini Linux that runs before the real one. Its job: find the real disk and hand over. It's a file (/boot/initrd.img-*) built from scripts in /etc/initramfs-tools. Something nasty got built into it.

Hint 2: get past it, then hunt

exit lets this boot carry on. Once you're in, the zombie's scripts are somewhere under:

ls /etc/initramfs-tools/scripts/*/
🩹 The fix (spoiler)

Type exit at (initramfs). Then kill the zombie and rebuild the initramfs (slow in a browser: a couple of minutes, let it chew):

sudo rm /etc/initramfs-tools/scripts/init-premount/zombie
sudo update-initramfs -u
📌 Make it stick

Deleting the script is only half of it: the old initramfs file still has the zombie inside until you rebuild. Then:

er-checkup

Why this happens to real servers:

Real life: a broken hook from a driver package, a bad cryptsetup or LVM setting, a half-finished kernel update. Rule: change anything that goes into the initramfs, rebuild it. And keep an older kernel around to boot from.

Patient 4: the imaginary disk

What the patient says:

Timed out waiting for device /dev/disk/by-label/ghost-vault. You are in emergency mode. Give root password for maintenance

Hint 1: who promised this disk?

/etc/fstab lists every disk to mount at boot. If one never shows up, systemd panics and stops. The root password on this machine is boo.

Hint 2: look
journalctl -xb | grep -i ghost
cat /etc/fstab
🩹 The fix (spoiler)

Log in with boo, make sure you can write, delete the lying line, reboot:

mount -o remount,rw /
nano /etc/fstab        # delete the ghost-vault line, Ctrl+O, Enter, Ctrl+X
systemctl reboot
📌 Make it stick

Back in? Ask the doctor:

er-checkup

Why this happens to real servers:

Real life: a USB drive got unplugged, a disk died, a UUID was mistyped. Two habits: add nofail to disks the system can live without, and run sudo mount -a (or sudo findmnt --verify) after editing fstab, before you reboot.

Patient 5: locked out of my own house

What the patient says:

Give root password for maintenance... Login incorrect. (The ghost changed it. Nobody knows the new one.)

Hint 1: who really owns a computer?

Whoever sits at its keyboard during boot. GRUB lets you change the kernel command line, and init=/bin/bash tells the kernel: skip everything, skip passwords, just give me a root shell.

Hint 2: how do I even reboot?

You can't log in, so use the power button: the ⏻ key under the screen. Then press an arrow key at the GRUB menu, and e.

🩹 The fix (spoiler)

⏻ to reboot. At GRUB press e, go to the end of the linux line, add rw init=/bin/bash, then Ctrl then x. (Plan B: press c and type it:)

linux /boot/vmlinuz-6.1.0-53-686 root=LABEL=possessed console=ttyS0 rw init=/bin/bash
initrd /boot/initrd.img-6.1.0-53-686
boot

At the # prompt:

mount -o remount,rw /
passwd root            # pick a new one
sed -i '/ghost-vault/d' /etc/fstab
sync
📌 Make it stick

No systemd is running, so reboot won't work: press ⏻. Then:

er-checkup

Why this happens to real servers:

Real life: the only admin left the company. Also the scary lesson: anyone at the console owns the box. That's why real servers lock GRUB with a password and encrypt their disks.

Same machine as the possessed PC, booted the real way through GRUB. Built with v86 (BSD), xterm.js (MIT) and Debian GNU/Linux. Sources: software and sources.