In The Outage That Wasn't I gave my router a second DNS resolver to fall back on, and proved it by killing the first. That fixed one path. Two weeks later, two false alarms one quiet morning showed me a second path I'd never tested, and the fallback on it had never worked, on any container, ever.
Two alarms, fifty seconds each
At 08:57 my uptime monitor reported my public website down. At 11:03 it reported my cloud down. Each alert cleared itself after exactly 50 seconds. Nothing was actually down. The error, at the monitor, was EAI_AGAIN: it couldn't resolve the name it was supposed to check.
The obvious next step was to ask the resolver what it had answered. It hadn't answered anything. Both lookups were missing from its query log entirely. That was the useful finding: a wrong answer would have been in the log. A missing entry meant the question never reached the resolver at all.
Before asking what DNS said, check whether DNS was ever asked.
Two nameservers, one of them imaginary
Most of my containers list two nameservers: the main resolver on the router, and the mirror on my backup node as the secondary. 16 out of 22 on the main node had it. It looks like textbook redundancy, and that's exactly why nobody had ever looked at it twice.
The mirror listens on port 5353, not 53. It has to: that's where the router's own DNS forwarder sends queries, which is the path I'd tested in the earlier post. But a container's system resolver, glibc, doesn't let you put a port on a nameserver line. It only ever asks port 53. Nothing was listening there.
So the secondary had never answered a single query, from any container, since they were built. Worse, it wasn't harmless dead weight. When one packet to the main resolver got lost, glibc did exactly what it's designed to do and tried the secondary, then waited out a five-second timeout against a closed port. On a monitor that checks every minute, that was enough to miss a beat and raise a false alarm.
The router-side failover I'd been proud of covered the router's path to its upstreams. The path from each container to the router was never covered. It only looked covered, because two lines were written in a file.
The fix that didn't touch the router
I could have moved the mirror to port 53, but then the router's forwarder config would need to change, and that's a file I've learned to leave alone. So the mirror's container got a small firewall rule instead: anything arriving on port 53 is redirected to 5353. The router keeps talking to 5353 as before; the containers' port-53 queries now land in the same place.
One detail nearly undid it. Adding the rule live works immediately, but that container's firewall file begins by flushing every rule and loading itself from scratch at boot. A rule added only at runtime would have vanished at the next reboot, and the fallback would have quietly gone back to not existing. The rule has to live in the file.
I also made the four monitors that check public names a little less jumpy: they now need two failures in a row before alerting. Even that needed checking at the source. The monitor's setting counts retries, so the value that means "two consecutive failures" is 1, not 2. Writing the number that read naturally would have asked for three.
Proving it from the outside
When I re-checked six days later, I didn't trust my notes. A dig to the mirror on port 53 got an answer, while the list of open sockets inside that container still showed nothing bound to 53. So the redirect, and only the redirect, was doing the work. As a control, the same query to two addresses that shouldn't answer DNS timed out, which ruled out something on the router quietly intercepting all DNS and making the test pass for the wrong reason.
The postscript about my own notes
I fixed this on the day, wrote it up in the notes for the monitor and the resolver, and skipped the step where an outage also goes into the incident log. The only trace left was a link to an incident page that had never been written. Six days later my weekly documentation audit flagged that broken link, and that's how this entry got its write-up. The link rot was the audit trail.
What I'd take from this
- A config line that names a host is not evidence anything answers there. Two nameservers were listed, one had never replied, and no dashboard could show that, because the file looked correct.
- A fallback you have never exercised is not a fallback. Test the second resolver by breaking the first, from the client that will actually use it.
- A missing log entry is evidence. "Never asked" and "answered wrong" live in different places, and the query log tells you which one you have.
- A runtime fix that a reboot erases is a fix with a timer on it. Put it where the boot path will load it.
Comments