Notes
Server dead or off the network?
How to tell a crashed machine apart from one that has just lost its network, and per
2 min read
When a server stops responding, the first question is always the same: is it dead or is it just off the network? The distinction changes everything: a dead machine needs someone to go to it in person; one that has lost its network can be fixed remotely. This week I had to live through it again, and I explain how I diagnosed it.
The two pings
There are two ways to ask a machine if it's there, and they don't say the same thing:
- Normal ping (ICMP): the good old ping (a brief signal that asks "are you there?" and waits for a reply). You're asking the network. If it doesn't respond, the machine might be off, without power, without a cable, or with the network down.
- Tailscale ping (the VPN I use: a private network that connects my machines remotely): you ask within the private network. If the machine appears as connected but nothing comes back, it means it's on but hung or without a physical network.
In my case: the machine appeared in the Tailscale list but neither the ICMP ping nor the Tailscale ping got anything back. Hundreds of packets (network messages), zero replies. Conclusion: the machine was completely dead. It wasn't a VPN or DNS (the name-to-address translator) problem.
The side effects: the mounts
This is the part nobody tells you: when a server dies, it dies with its own things, but it also hangs other people's things.
I had a remote filesystem mounted via SSHFS (so a folder on another server appears as local, over SSH) from another server to the dead machine. When it died, the mount got stuck, and any process that touched that path froze without error or warning: neither ls, nor du, nor automation scripts (like the A disk watchdog with cron). Zombie processes (that no longer do anything but don't close) looking the other way.
How to clean it up
It gets cleaned up live, without turning anything off:
# unmount live a hung mount
fusermount -uz /ruta/del/muntatge
# kill the hung sshfs processes
pkill -f "ruta/del/muntatge"
The lesson: when a server goes down, the first thing you should do not do is try to repair it, but rather check which other machines had mounts or connections to it. Collateral damage often annoys more than the original crash. And that machine's disk had already given me trouble before: The disk that filled itself up.
The result
The machine recovered by physically rebooting it (the only way when it's really hung), and the mounts worked again after remounting them. But that afternoon I learned a lesson I had read a thousand times and had never experienced with such clarity: the network is a system, and when one piece falls, the others notice too.