The Outage That Wasn't: when my own tools lied to me

I set out to remove a single point of failure and, for about fifteen very unpleasant minutes, believed I had created a total one instead. The twist — which I only found once I stopped panicking and picked up a different tool — is that nothing was ever actually broken. My instruments were lying to me, and I'd taken real, damaging action against a perfectly healthy system on the strength of a false reading.

The thing I was fixing

All DNS in my house flows through one ad-blocking resolver on the router. It's a lovely setup right up until that one resolver dies — at which point the entire apartment loses the ability to turn names into addresses, and everything appears "offline" even though the internet is perfectly fine. There's a second resolver on another box, a full mirror of the first, but nothing fails over to it. It just sits there.

The fix is small and elegant: point the router's DNS forwarder at both resolvers. If the local one stops answering, queries fall through to the backup, and the house never notices. Wire it up, then test it the only way that counts — kill the local resolver and confirm names still resolve through the mirror.

The test that went wrong

So I killed the local resolver and ran my check. It came back FAILING. Nothing resolved. I tried again — still failing. Then I tried to restart the resolver and it wouldn't come back. My check said it was down; the service manager insisted it was running; a process check reported zero copies of a program the service manager swore was alive. Nothing agreed with anything, and the one story they seemed to tell in chorus was: you have just taken DNS down for the whole house.

This is the worst moment in any debugging session — the one where you believe you've broken something, and you start doing things.

And I did things. I restarted the resolver a few more times, harder. I killed processes. And in doing so I actually did cause a small mess — a couple of half-started instances stepping on each other — a real problem, created entirely in response to an imaginary one. The classic failure cascade: a false alarm provokes a real intervention that provokes a real fault.

The moment it turned

I stopped. Took a breath. And did the one thing I should have done first: I used a different tool. Instead of the router's built-in conveniences, I asked two questions with two instruments I actually trust — netstat for "is something listening on the DNS port?" and dig, from my laptop, for "does a query actually get answered?"

Both said: everything is fine. One clean resolver, listening exactly where it should, answering every query I threw at it. The DNS had been working the entire time.

What had actually been lying

The router runs BusyBox — a single tiny binary that stands in for dozens of Unix tools, trading corners for size. And two of those corners had quietly wrecked my afternoon.

The process check, pgrep -x, reported zero instances of a resolver that netstat could plainly see listening and answering. Its counting flag, pgrep -c, isn't even implemented there — it printed an error, and I'd read that error as "zero". And my quick one-liner to test resolution piped BusyBox's nslookup into an awk that grabbed the wrong column, because BusyBox formats its answer differently than the tool I'd built the habit on. So every successful lookup printed a triumphant "FAILING".

# what I believed          # what was actually true
pgrep: 0 instances    →    netstat: listening, one clean process
nslookup|awk: FAILING →    dig: answered, correct address, every time

Three tools, three lies, all pointing the same way — which is exactly what made them so convincing. A single weird reading you question. A unanimous chorus of weird readings you believe.

The test, done properly

Once I trusted the right instruments, the actual test was clean and quietly satisfying. Kill the local resolver — confirmed genuinely dead by netstat (nothing on the port) and by dig (connection refused). Then ask the router's forwarder for a fresh, uncached name, and watch it come back resolved through the mirror. Seamless failover, exactly as designed. Restart the local resolver, and it takes back over. It had worked from the very first attempt; I'd just been reading a broken gauge.

There was even one real bug hiding under the imaginary one. My first instinct for the config was to tell the forwarder to try the servers in strict order — local first, mirror second. That version genuinely did break failover: with the dead server listed first, the forwarder sat there timing out instead of falling through. The setting that actually worked was the simpler one — query both in parallel, take whichever answers first — which fails over instantly because the survivor is always already in the race. The obvious knob was the wrong knob, and I'd never have separated the real bug from the fake one if I hadn't first proven the system was healthy.

The lesson

The big one: the most dangerous instrument in debugging is a broken instrument, because a false failure reading doesn't make you cautious — it makes you act. You start restarting and killing and "fixing", and now you're introducing real faults to chase a phantom. Before you believe a system is broken, spend the ten seconds to confirm your tool isn't.

And the practical corollary, for anyone poking at a constrained box: on BusyBox, the convenience tools are not the ones you reach for by reflex. Verify with the primitives. netstat for "is it listening", dig for "does it answer", and a fresh uncached name so the cache can't lie to you either. Everything else on that little router is allowed to be wrong, and on that particular afternoon, three separate things were — all at once, all in the same direction, all completely full of it.