ArunNetworkingPro
👻

Ghost Cam: catch a haunting on Grafana

Ghosts hate graphs. Graphs have a memory.

Something keeps making your server slow at odd hours, and by the time you log in, it's gone. Ghost, or Linux? Real sysadmins don't guess. They set up a camera: Prometheus records the server's vital signs every few seconds, and Grafana draws them. Then the graphs remember what happened while nobody was watching. Let's build one, release a (harmless) ghost, and catch it on tape.

Intermediate45 minutesMonitoring

🧰 What you need

Let's build it

1

Download the ghost cam

curl -O https://arunnetworkingpro.com/labs/ghost-cam.sh
less ghost-cam.sh

Read it before you run it. The top of the script lists everything it touches and how to undo it. Press q to leave less.

2

Start recording

bash ghost-cam.sh

It starts three containers: node_exporter (reads the server's CPU, memory and disk), Prometheus (writes them down every 5 seconds) and Grafana (draws the graphs). The first run downloads the images, so give it a few minutes. At the end it prints a Grafana password: copy it.

Port 3000 already taken? GHOST_CAM_PORT=3001 bash ghost-cam.sh, and use 3001 everywhere below.

3

Open the camera

In a browser on the same machine, go to http://localhost:3000. Log in as admin with the password the script printed (it's also in ~/ghost-cam/.grafana-password). Then open Dashboards → 👻 Ghost Cam.

Running this on a server with no screen? From your laptop, ssh -L 3000:localhost:3000 you@server-ip (-L forwards your laptop's port 3000 through SSH to the server's), then open http://localhost:3000 on the laptop. Or set it up with GHOST_CAM_LAN=1 bash ghost-cam.sh so Grafana listens on your whole home network.

Watch it for a minute or two. This is what normal looks like on your machine. Remember it: you can't spot something weird until you know what's normal.

4

Release the ghost

bash ghost-cam.sh haunt

Over the next ~17 minutes, the ghost strikes 3 times, in secret, at random. Keep the Ghost Cam open and grab a notepad. For every strike, write down when it started, how long it lasted and which graph it hit.

No peeking with top for now. The camera is your only witness.

5

Find what it left behind

When things go quiet, look at Disk used. One graph never came back down. The ghost hid something in your home folder. Hunt it down:

find ~ -type f -mmin -60 -size +20M 2>/dev/null -exec ls -lh {} +

In plain words: search your home folder (~) for files (-type f) changed in the last 60 minutes (-mmin -60) that are bigger than 20 MB (-size +20M), hide the "permission denied" noise (2>/dev/null), and show each one with its size (-exec ls -lh {} +).

Found a suspicious file? Delete it with rm and watch the disk graph drop a few seconds later.

✅ How you know it worked

Time for the truth. Read the ghost's diary and compare it with your notes:

bash ghost-cam.sh reveal

You win if you got all 3 attacks, in the right order, within about a minute each, and you deleted the ghost's file. Ghost 0, you 1.

💥 Break it on purpose

Let it haunt you again, but this time put the camera away and use only the terminal:

bash ghost-cam.sh haunt
top        # q to quit
free -h
df -h

Which was easier: the terminal or the graphs? Now imagine the ghost struck at 3 a.m. while you were asleep. Only one of them remembers.

Want the ghost gone right now? bash ghost-cam.sh stop. Done with the lab? bash ghost-cam.sh clean removes the containers, the files and any ghost leftovers.

The ghost cam doesn't come back after a reboot. Run bash ghost-cam.sh again to switch it on. clean also prints the command to delete the downloaded images.

🧠 What's really going on

Linux keeps live counters for everything in /proc and /sys: how long each CPU core has been idle, how much memory is free, how full every disk is. node_exporter reads those counters and serves them as plain text. Prometheus collects that text every 5 seconds and stores each number as a time series: a value with a timestamp. Grafana asks Prometheus questions in a language called PromQL and draws the answers.

The CPU graph, for example, asks: 100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[30s]))). In plain words: "over the last 30 seconds, how much of the time were the CPUs not idle?" Click Edit on any panel to see its question, then change it and see what happens.

These are the same kind of tools big teams use to watch their servers, just tiny. Next level: add Loki, and you can catch the ghost's whispers in the logs too.

🎉 Got it, thank you!

Your comment just landed in my inbox. I read every one, and I'll reply by email.

🤔 That didn't go through

Something in the form looked off. Check your name, email and comment and try again, or just email me.

🐢 Whoa, slow down

That's a lot of comments in a short time, so the box is taking a breather. Try again later, or email me.

😴 The comment box is napping

My server is taking a quick break, so your comment couldn't be sent. Sorry! Please email me instead.

💬 Leave a comment

Stuck, found a better way, or just built it and want to brag? Tell me. It comes straight to my inbox (nothing is posted publicly), and I'll reply by email.

Your email is only used to reply to you. Never shared, never added to any list.

← Back to all Linux labs · Stuck? Email me