The disk that filled itself

Vaig passar una tarda sencilla mirant un servidor que deia tenir el disc ple... i que tenia menys de sis gigas de fitxers reals. df (l'eina que diu quant espai lliure queda a cada disc) assegurava que tot estava ocupat; du (la que suma el que pesen els fitxers) no trobava la meitat. Aquesta és la història del misteri i del truc que el va resoldre.

I spent a quiet afternoon looking at a server that claimed its disk was full... and that had less than six gigabytes of actual files. df (the tool that reports how much free space is left on each disk) insisted everything was used up; du (the one that adds up the weight of the files) couldn't find half of it. This is the story of the mystery and the trick that solved it.

The symptoms

• df -h said: 100% used, zero available.
• du -sh /home summed up far less than half the disk.
• Databases started failing because they couldn't write.
• And the best part: no large files, no suspicious folders, no visible explanation.

When df and du disagree, something very specific is going on. It's not magic: something is hiding files from the tools.

The three classic causes

1. Deleted files that a process still has open

A process (a running program) writes to a file, someone deletes it, and the process keeps writing. The file shows up nowhere, but it takes up space until the process dies.

2. Mounts covering folders

A mount is connecting a disk to a folder on the system. If there is data written to /home/dades and then you mount a disk on top of it, that data nobody sees it — like a drawer that ends up behind a new wardrobe — but it still takes up space.

3. File systems with broken accounting

Rarely, the file system itself loses track of what's there. Time for fsck (the tool that checks and repairs), with the disk unmounted.

The trick: looking without the mounts

My case was number 2, with a twist. The phantom space lived under the mount point of a service that downloads content continuously. With the mount active, nothing is visible; when it drops, the data appears. And I couldn't run sudo (execute commands as administrator), so I looked at the disk with an ephemeral Docker container (a mini-environment you spin up, do the job, and it disappears):

docker run --rm -i -v /home:/hosthome:ro node:22-slim sh -s <<'EOF'
du -xsh /hosthome/fabricante/mnt/* 2>/dev/null
find /hosthome -xdev -type f -printf "%s\n" | awk '{s+=$1} END {print s/1073741824" GB"}'
EOF

The container doesn't have my user mounts: it sees the disk as it really is. There appeared the gigabytes that du couldn't find.

The resolution

• The offending service: disabled.
• The space: recovered, from zero free to fifteen gigabytes.
• And a disk monitor that alerts when it fills up (the one from A disk monitor with cron) so it doesn't sneak up on me again.

The lesson I take away: disk tools assume a world without mounts. When the numbers don't add up, don't argue with du: look for what's underneath.