Linux+ Break Room · Case 09
Things that go bump at midnight
The midnight jobs never ran. The spell book has been tampered with. Nobody saw a thing.
Every minute, cron is supposed to knock once and light a lantern. The heartbeat log is empty, and the lantern log only says "not found". A systemd timer should run a ritual at midnight: it has never fired, not once. And the grimoire, a git repository of spells, has been quietly changed. A secret was committed and "removed", a spell was broken, and a protection charm was deleted. The history remembers everything. Make the house quiet again.
🧰 What you need
- Any Linux with cron, systemd and git (
sudo apt install -y cron gitif they're missing). - No sudo. It adds two lines to your own crontab (each ending in
# breakroom-09; your other lines are never touched), two systemd user units, and a folder.--cleanremoves all of it.
Get the case
Rule 4: read it before you run it, less breakroom-mission-09.sh. The top of the file lists everything it creates and how --clean removes it.
curl -O https://arunnetworkingpro.com/labs/breakroom-mission-09.sh
bash breakroom-mission-09.shAt any point, ask the house how many ghosts are left:
bash breakroom-mission-09.sh --checkPart A: cron
The heartbeat that never beats
crontab -l
ls ~/breakroom/mission-09/cronThe knock job runs every minute, yet its log file never appears. Find out what cron actually ran. Then fix it.
Hint
cron writes down each command it starts: sudo journalctl --since -5min | grep CMD (or grep CRON /var/log/syslog). Look at where the command stops. In a crontab, % means "newline", so it has to be escaped as \%. Edit with crontab -e.
Command not found
cat ~/breakroom/mission-09/cron/lantern.log
~/breakroom/mission-09/bin/lantern-checkIt works when you run it. cron can't find it. Why? Fix the job.
Hint
cron runs jobs with a tiny PATH (usually just /usr/bin:/bin), not yours. Use the full path to the program in the crontab line, or set PATH= at the top of the crontab.
Part B: the midnight timer
A time that doesn't exist
systemctl --user start breakroom-09-midnight.timer
systemctl --user status breakroom-09-midnight.timerFind what's wrong with the schedule, fix it so it fires every night at midnight, and make it survive a reboot.
Hint
systemd-analyze calendar '...' tests any schedule and shows the next time it would fire. Clocks go from 00:00 to 23:59. To enable it, the timer needs [Install] with WantedBy=timers.target. Then daemon-reload, enable --now, and systemctl --user list-timers. (User timers only run while you're logged in, unless loginctl enable-linger $USER is on.)
Don't wait for midnight
Run the ritual right now, once, to prove it works.
Hint
A timer only triggers its service. You can start the service yourself: systemctl --user start breakroom-09-midnight.service, then cat ~/breakroom/mission-09/ritual.log.
Part C: the grimoire's history
The secret that was "removed"
cd ~/breakroom/mission-09/grimoire
git log --onelineSomeone committed a secret, then deleted it. What was the secret?
Hint
Deleting a file in a new commit doesn't erase it from history. git log --oneline --all -- .env lists commits that touched it. git show <commit>:.env prints the file as it was in that commit.
Who broke the spell?
bash spell.shFind the commit that broke spell.sh, and who made it. Undo it without rewriting history.
Hint
git log -p -- spell.sh shows every change to one file, with its author. git revert --no-edit <commit> adds a new commit that undoes it (--no-edit skips opening an editor). (git bisect can hunt it down automatically in bigger histories.)
Bring back the protection
protection.txt was deleted. Bring it back from history, and commit it.
Hint
git log --oneline --diff-filter=D -- protection.txt finds the commit that deleted it. The file still exists in the commit before that one: git checkout <commit>^ -- protection.txt, then git commit -m "Bring back protection".
✅ How you know you won
bash breakroom-mission-09.sh --check says Verdict: 6/6. Nothing goes bump anymore. It was Linux all along. Case closed. (The cron checks need a minute or two after your fix.)
Answer key (no peeking until you've tried)
A1: cron ran echo "knock at $(date + and stopped: the % cut the line. Write date +\%H:\%M. A2: lantern-check isn't on cron's PATH; use $HOME/breakroom/mission-09/bin/lantern-check (the full path). B1: OnCalendar=*-*-* 24:00:00 is invalid; use *-*-* 00:00:00 (or daily), add [Install] / WantedBy=timers.target, daemon-reload, enable --now. B2: systemctl --user start breakroom-09-midnight.service. C1: summon_key=candle-7734-agatha (commit 2136ba9, "Add summoning config"). C2: 0d0e70f "Tidy up spells" by Agatha (exti 0); git revert --no-edit 0d0e70f. C3: deleted in a2b4668; git checkout a2b4668^ -- protection.txt and commit.
💥 Let it haunt you again
Run the script again. The haunting starts over, fresh:
bash breakroom-mission-09.shNow do it without the hints. And when you're done for good: bash breakroom-mission-09.sh --clean.
🧠 The ghost, explained
Verdict: not a ghost. The midnight bumps were a percent sign, a PATH, an impossible time and a git history that remembered everything.
cron is not your shell. It runs jobs with almost no environment: a tiny PATH, no aliases, no .bashrc. Full paths and explicit logging (>> file 2>&1) save hours. And % is special in crontabs, a detail that has broken countless date commands.
systemd timers are cron's modern cousin. A .timer starts a .service on a schedule. OnCalendar= uses its own format, which systemd-analyze calendar checks for you. Persistent=true runs missed jobs after the machine was off, and list-timers shows when everything fires next.
git never forgets. A secret that was ever committed is in the history until someone rewrites it, and every clone has a copy. The real fix for a leaked secret is to change the secret itself.
Undo with new commits. git revert adds a commit that reverses an old one, so everyone else's history stays valid. git checkout <commit> -- file (or git restore --source) brings back any file from any point in time.
← Case 08 · All cases · Case 10 → · Stuck, or found a better way? Email me
🎉 Got it, thank you!
Your comment just landed in my inbox. I read every one, and I'll reply by email.
🤔 That didn't go through
Something in the form looked off. Check your name, email and comment and try again, or just email me.
🐢 Whoa, slow down
That's a lot of comments in a short time, so the box is taking a breather. Try again later, or email me.
😴 The comment box is napping
My server is taking a quick break, so your comment couldn't be sent. Sorry! Please email me instead.
💬 Leave a comment
Stuck, found a better way, or just built it and want to brag? Tell me. It comes straight to my inbox (nothing is posted publicly), and I'll reply by email.
Your email is only used to reply to you. Never shared, never added to any list.