⚰️

The thing that won't stay dead

It rises. It falls. It rises again. systemd is losing patience, and so are you.

Something in this server knocks. A little background service called poltergeist is supposed to knock once every five seconds, forever, so everyone knows the house is alive. Tonight it won't start. When it does start, it dies. It rises again, dies again, until systemd gives up on it. And after a reboot, nothing stirs at all. Bring it back, keep it back, and make it rise on its own.

Case 02 of 10Services & usersTroubleshootingNo sudo

🧰 What you need

Any Linux that runs systemd: Ubuntu, Debian, Fedora, a Pi, a VM. Most do.
On WSL, new Ubuntu installs already run systemd. Check with ps -p 1 -o comm=: it should say systemd. If not, add [boot] and systemd=true to /etc/wsl.conf (that one edit needs sudo), then run wsl --shutdown from Windows.
A text editor you're happy in (nano is fine).
No sudo. The poltergeist is a user service: it runs as you, so your real system services are never touched.

Get the mission

Read it first: less breakroom-mission-02.sh. It touches exactly two things: one folder (~/breakroom/mission-02) and one unit file (~/.config/systemd/user/poltergeist.service). When you're done, bash breakroom-mission-02.sh --clean sends the poltergeist to rest and removes both.

curl -O https://arunnetworkingpro.com/labs/breakroom-mission-02.sh
bash breakroom-mission-02.sh

Part A: raise it

Raise it

systemctl --user start poltergeist

It refuses. Find out why: which line of the unit file is systemd unhappy about?

Hint

systemctl --user status poltergeist shows the last few log lines, and they name the exact line number. For the full story, try journalctl --user-unit poltergeist.

Fix the path

Open ~/.config/systemd/user/poltergeist.service in your editor and fix the line systemd complained about. Then tell systemd you changed it, and try to raise it again.

Hint

systemd wants an absolute path, one that starts with /. You can use %h for your home folder. After any edit to a unit file, run systemctl --user daemon-reload.

It rose... and fell

This time start doesn't complain, but the poltergeist isn't running. Dig through the logs and find the one line that says what's actually wrong.

Hint

Scroll past the systemd noise ("Scheduled restart job", "Failed with result"). The poltergeist itself left a message. If journalctl shows nothing on your box, it also writes to ~/breakroom/mission-02/spirit/spirit.log.

Count the attempts

Before you fix anything: how many times did systemd raise it before it gave up? One command, one number.

Hint

systemctl show lists every property systemd keeps about a unit. -p picks just one. The one you want counts restarts.

The missing name

Every ghost needs a name. The poltergeist needs GHOST_NAME, and the value is sitting in a file right there in the spirit folder. So why can't the service see it? Fix it, and let it knock.

Hint

Compare the EnvironmentFile= line with ls ~/breakroom/mission-02/spirit. And what does the - right after the = do? If systemd still says "repeated too quickly", run systemctl --user reset-failed poltergeist first.

Part B: break the curse loop

Only the bad news

Show only the error lines for this service. None of the chatter.

Hint

journalctl can filter by priority with -p. The poltergeist tags its fatal message as an error (priority 3). systemd's own "Failed to start" line may show up next to it, but the poltergeist's line is the one that says why.

Change the rules without touching the file

Restarting every 100 milliseconds is just panicking. Make it restart only when it fails, and wait 5 seconds first. But don't edit the original unit file: add an override on top of it.

Hint

systemctl --user edit poltergeist opens an empty override file. Put your settings under a [Service] heading. Check the result with systemctl --user cat poltergeist, and restart it to use it.

Crash it on purpose

systemctl --user kill -s KILL poltergeist

Watch it: does it rise again, and how long does it take? What's the restart count now?

Hint

watch systemctl --user is-active poltergeist updates every 2 seconds. Being killed by a signal counts as a failure, so on-failure brings it back.

Part C: make it rise after a reboot

Make it rise after a reboot

Try systemctl --user enable poltergeist. systemd prints a warning and enables nothing. Read what it says, add what's missing to the unit file, and enable it.

Hint

It needs an [Install] section saying what should pull it in at startup. For user services, that's WantedBy=default.target. Don't forget daemon-reload.

Prove it

Show that it's enabled, then find the symlink that enable just created. Where does it live, and where does it point?

Hint

is-enabled answers the first part. enable printed the second part as it worked. Or go looking with ls -l ~/.config/systemd/user/default.target.wants/.

✅ How you know you won

systemctl --user is-active poltergeist says active, and is-enabled says enabled.

systemctl --user show poltergeist -p Restart -p RestartUSec says on-failure and 5s.

tail -f ~/breakroom/mission-02/spirit/spirit.log shows a fresh knock every five seconds. Spooky, but healthy.

Answer key (no peeking until you've tried)

A1: line 8, ExecStart is not an absolute path. A2: ExecStart=%h/breakroom/mission-02/spirit/haunt.sh, then daemon-reload. A3: FATAL: GHOST_NAME is not set. Nobody to haunt with. A4: systemctl --user show poltergeist -p NRestarts gives 5. A5: the file is spirit.env, not spirit.conf. The - hid the missing file. B1: journalctl --user-unit poltergeist -p err. B2: [Service] with Restart=on-failure and RestartSec=5s. B3: back in about 5 seconds, restart count 1. C1: add [Install] and WantedBy=default.target. C2: default.target.wants/poltergeist.service, pointing at your unit file.

💥 Let it haunt you again

Run the script again. The curse is back, fresh, and your override is gone too:

bash breakroom-mission-02.sh

Now do it without the hints. Then try it the system way on a spare VM: same unit, but in /etc/systemd/system/ with sudo systemctl and journalctl -u.

🧠 The ghost, explained

Verdict: not a ghost. The thing that wouldn't stay dead was systemd doing exactly what the unit file told it to. The curse was four small mistakes in one file.

A unit file has three rooms. [Unit] says what it is and when to start it. [Service] says how to run it. [Install] says what should pull it in when you enable it. No [Install], nothing to enable.

systemd doesn't guess paths. ExecStart needs an absolute path (or a plain program name it can find on its PATH). %h is systemd's shorthand for your home folder. And after every edit to a unit file, daemon-reload, or systemd keeps using the old copy it has in memory.

That little dash is sneaky. EnvironmentFile=- means "if this file is missing, don't complain". Handy for optional settings. Terrible when the file you needed has a typo in its name.

Restart=always isn't a fix, it's a treadmill. A service that dies straight away just dies again and again, until the start limit (StartLimitBurst tries within StartLimitIntervalSec) gives up with "Start request repeated too quickly". Find the real error first. Then reset-failed clears the penalty box.

Never edit the original, use a drop-in. systemctl edit writes an override.conf next to the unit. Package updates replace the original file, but your override survives. systemctl cat shows both, stacked in order.

Start and enable are different things. start means now. enable means next time too. enable --now does both. User services start when you log in and stop when you log out. To start them at boot with nobody logged in, look up loginctl enable-linger.

User vs system. Everything here works the same for real system services: drop the --user, add sudo, keep units in /etc/systemd/system/, and read logs with journalctl -u. Same on Debian, Ubuntu, Fedora and Red Hat: they all run systemd.

← Mission 01 · Back to the Break Room · Stuck, or found a better way? Email me

🎉 Got it, thank you!

Your comment just landed in my inbox. I read every one, and I'll reply by email.

🤔 That didn't go through

Something in the form looked off. Check your name, email and comment and try again, or just email me.

🐢 Whoa, slow down

That's a lot of comments in a short time, so the box is taking a breather. Try again later, or email me.

😴 The comment box is napping

My server is taking a quick break, so your comment couldn't be sent. Sorry! Please email me instead.

💬 Leave a comment

Stuck, found a better way, or just built it and want to brag? Tell me. It comes straight to my inbox (nothing is posted publicly), and I'll reply by email.

Your email is only used to reply to you. Never shared, never added to any list.