Systems
A disk watchdog with cron
Monitor server free space with a script and cron: silence
2 min read
For a while now, a question has been on my mind: how to monitor a server without setting up a full corporate monitoring system? There are solutions everywhere, but for a home server they feel like using a cannon to kill a fly. My answer: cron (the system task scheduler) and a script.
The specific problem
On fxserver, the disk where /home lives fills up quickly when some automation goes haywire — I already told those scares in The disk that filled itself. A full disk is a silent alarm: databases stop writing, logs (the system journal) grow uncontrollably, and in the end nothing works.
The idea
The program is minimal: check free space, clear caches if needed, and notify only if after cleaning the problem persists.
- If there's plenty of space: silence. It doesn't bother you.
- If there's little left: first try to clean regenerable things (caches that reload on their own).
- If after cleaning there's still little left: clear warning, with the suspected cause.
Silence is the most important part of the design: if it only speaks when there's a problem, when it speaks you pay attention.
Why cron and not a service
Cron runs on its own, every 30 minutes, for months now, and has never asked for anything in return. It doesn't need daemons (programs that live in the shadows providing services), nor dashboards, nor anything to maintain: a bash script and a line in the crontab, the file where cron notes down its tasks. For the size of my project, this balance was exactly what was needed.
The detail reality has taught me
The first version oscillated between silence and useless alarms. I learned that "2 GB remaining" is not the same as "I cleaned and it keeps dropping": the latter is a real leak, the former is normal maintenance. Distinguishing them is the difference between a useful watchdog and one that ends up being ignored.
And a lesson that cost more: when something is written to disk under a mount point (the folder where you connect a disk or partition) that is later hidden, du — the tool that measures what each folder occupies — doesn't see it but the disk does notice. Tools deceive you if you don't know what's underneath.