escandell.

Notes

hpServer: the Wi-Fi that kept dropping and how it heals itself

Complete diagnosis of a Realtek RTL8822BU adapter that was losing its link

6 min read

A few days ago my server went offline on its own. Suddenly, and for no apparent reason, it stopped responding: I couldn't get into it from my computer, the services that depended on it hung, and for a while it seemed the machine had died.

It wasn't dead. It was still on and running. The only thing failing was its network connection: a small Wi-Fi adapter connected via USB. This is the story of how I figured it out and how I set up a system that repairs itself.

What is hpServer and why I care about it

hpServer is an old computer converted into a home server. It runs internal home services —storage, tests, automations— and I access it from anywhere: via SSH (the way to log into a remote computer as if you were sitting in front of it) and via Tailscale (a private network that connects all my devices as if they were in the same room). A note in case anyone is wondering: this website you're reading doesn't live here, but on fxserver, another of my servers. When hpServer goes down, several things at home stop working at once, and it always seems worse than it is.

How to figure out what failed: thinking in layers

When the internet fails, the blame can lie in many places: the computer, its adapter, the router, the provider... A connection is like a chain: if one link breaks, everything that comes after stops working. The trick is to find the first broken link, because that's the only one that needs fixing.

During an outage I checked the Wi-Fi status and saw this:

wlx6c5ab074f239: <NO-CARRIER,BROADCAST,MULTICAST,UP>
state DOWN
Not connected.

In plain terms: the system was asking the adapter "connect" and the adapter couldn't. The ping to the router (an "are you there?" used to check whether a machine answers) got no response either. With that I ruled out everything that came next: the router, the provider, my private network... The problem was the first link: the adapter.

Crossing off suspects

Even so, I kept ruling things out one by one:

  • Power saving: I already had it disabled (sometimes the system "puts to sleep" the adapters to save power).
  • The USB port: the problem repeated itself on different ports.
  • The signal: good, with plenty of strength.
  • The system software: up to date. I even tried a newer version of the whole system. The problem stayed the same.

The decisive clue

The system keeps a log of everything that happens (the famous "logs"). In that log, every outage left the same message: the adapter asks a mini-program living inside it (the "firmware") for an acknowledgment and gets no response. It's like talking to someone who has fallen asleep.

And here the important discovery: restarting the connection did nothing. What worked was turning the adapter's software off and on again (the "driver": the program that acts as a translator between the system and the device). Like turning your home router off and on, but only for the adapter.

The solution: an automatic watchdog

Since the problem would happen again, I wanted the server to heal itself. So I set up a "watchdog", literally a guard dog: a small program that every minute asks itself a very simple question —is there network?— and if the answer is no twice in a row, it turns the adapter off and on. In a minute or two, the connection comes back.

This is the script it follows (the idea is in the paragraph above; the details are in the code):

#!/bin/bash

IFACE="wlx6c5ab074f239"
ROUTER="192.168.0.1"
MODULE="rtw88_8822bu"
LOG_TAG="wifi-watchdog"

# Si el router respon, tot va bé.
if ping -I "$IFACE" -c 2 -W 2 "$ROUTER" >/dev/null 2>&1; then
    exit 0
fi

logger -t "$LOG_TAG" "Wi-Fi sense connexió. Esperant 10 segons abans de confirmar..."

# Evitar actuar per un microtall.
sleep 10

if ping -I "$IFACE" -c 2 -W 2 "$ROUTER" >/dev/null 2>&1; then
    logger -t "$LOG_TAG" "La connexió s'ha recuperat sola."
    exit 0
fi

logger -t "$LOG_TAG" "El Wi-Fi continua caigut. Reiniciant el mòdul $MODULE..."

ifdown --force "$IFACE" >/dev/null 2>&1 || true
modprobe -r "$MODULE"
sleep 2
modprobe "$MODULE"
sleep 5
ifup "$IFACE" >/dev/null 2>&1 || true
sleep 10

if ping -I "$IFACE" -c 2 -W 2 "$ROUTER" >/dev/null 2>&1; then
    logger -t "$LOG_TAG" "Wi-Fi recuperat correctament."
    exit 0
else
    logger -t "$LOG_TAG" "ERROR: no s'ha pogut recuperar el Wi-Fi."
    exit 1
fi

Notice one detail: the ping goes to the router, not to the internet. If one day the internet goes down for the whole house, the watchdog does nothing: it only acts when the problem is internal, which is the only thing it can fix.

To run it automatically every minute I use the system task scheduler (systemd), with two very short files:

# /etc/systemd/system/wifi-watchdog.service
[Unit]
Description=HPServer Wi-Fi watchdog
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/wifi-watchdog.sh

# /etc/systemd/system/wifi-watchdog.timer
[Unit]
Description=Comprovar Wi-Fi de HPServer cada minuto

[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
AccuracySec=5s
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now wifi-watchdog.timer

The test

Setting up a self-repairing system and blindly trusting it is a bad idea. So I ran a test: I deliberately turned off the connection and waited without touching anything.

sudo ifdown --force wlx6c5ab074f239

In just over a minute, the watchdog detected it, reloaded the adapter, and the network came back. The system log kept the complete sequence:

Wi-Fi sense connexió. Esperant 10 segons abans de confirmar...
El Wi-Fi continua caigut. Reiniciant el mòdul rtw88_8822bu...
Wi-Fi recuperat correctament.

When everything is fine, the watchdog stays quiet: if it says nothing, everything is fine.

For next time

# està connectat el Wi-Fi i amb quina força?
sudo /usr/sbin/iw dev wlx6c5ab074f239 link

# els errors de xarxa del dia (el "diari" del sistema)
sudo journalctl -k -b --no-pager | grep -Ei 'rtw|8822|wlx|firmware' | tail -100

# què ha fet el vigilant
sudo journalctl -t wifi-watchdog --since today

# recuperar el Wi-Fi a mà (el mateix que fa el vigilant)
sudo modprobe -r rtw88_8822bu && sleep 2 && sudo modprobe rtw88_8822bu && sleep 5 && sudo ifup wlx6c5ab074f239

What I take away

When something "dies" at home, it's almost never the whole device: it's a single link in the chain. And many times you can build something that fixes it while you sleep. This time the link was a twenty-euro Wi-Fi adapter; the watchdog, half an hour of work.

And the saga continued: in the next note, an old router from a drawer ended up working in reverse so the server would stop depending on Wi-Fi in day-to-day use.