My homelab has an orchestrator: a program that knows every container, what runs inside it, and what depends on what. It can check a service's health and, with my approval, restart it. One evening I gave my dashboard a Diagnose button that asks the orchestrator for a read-only health check. I tried it on the container that runs my house's speech recognition, which I knew was working because it had transcribed my voice the night before. It said unhealthy.
A naming detail, or so I thought
The check's report was honest about what it had looked at. The container was running, the network port was open, and the system service it had checked was "inactive (dead)". It had checked for a service named voice, which is the container's name. The services actually running inside it are called wyoming-whisper and wyoming-piper.
My first note said "probably a naming detail in the orchestrator's map" and moved on. The next morning I looked properly.
The map knew the right name
The map already said wyoming-whisper. Every service entry can carry the real name of the program it runs, precisely because container names and program names often differ. The restart workflow read that field. The health check didn't: it always used the container's hostname, and nobody had noticed, because nothing had ever asked the health check about a service with a different name and then looked at the answer.
So I asked it about all of them. Ten running containers came back unhealthy: the reverse proxy, the orchestrator itself, my AI assistant's listener, the backup DNS server, the backup monitoring and notifications, a scraper helper and a couple of web servers. I checked each one by hand. In every case the container-named service didn't exist, and the real one was running fine.
Two halves of the same program had different opinions about what a service is called. One of them was right.
Then the map itself
Fixing the health check to read the right field made those ten pass. But it raised an obvious question: is the field itself ever wrong? So I swept every one of the 34 services in the map that lives in a container, and asked the container directly whether the named program was actually running.
Three weren't. One entry named its program slightly wrong: the real one has an extra word on the end. Two others, both in the camera's vision container, had no name at all and fell back to the container's hostname, which matched nothing. Those three were wrong for the restart workflow too. If any of them had ever needed a restart, the system would have dutifully restarted a program that doesn't exist and reported back.
The only other service not running was the one that's switched off on purpose.
Why nothing had caught it
This is the part worth keeping. The map is checked in plenty of ways: its structure, its dependencies, its backups, against what Proxmox says exists. None of those checks ever asked the one question that matters to a health check: is the thing you name actually there, running, under that name?
And the failure stayed invisible because of where the lie sat. A false "unhealthy" on a button nobody pressed costs nothing, until the day a dashboard trusts it, or an automation acts on it.
The fix, and proving it
Three changes:
- The health check now uses the service's declared name, the same way restart always did. For a container, it uses the declared name when exactly one service lives there.
- The three wrong map entries are corrected.
- A new section in my weekly audit asks every container whether every named service is actually running. It treats an unreachable machine as "skipped, not checked", never as "fine".
To prove the audit works, I ran it against the previous version of the map, the one with the three mistakes. It flagged exactly those three. Against the current map it reports 33 running and zero mismatches. An audit that has never been seen catching something is a hope, not a check.
What I'd take from this
- When two parts of a system describe the same thing, check that they agree. Here, restart and health check each had their own idea of a service's name.
- "Unhealthy" is a claim about something specific. Read what was actually checked before believing the verdict, in either direction.
- Verify the map against the territory, not just against itself. A map can be perfectly consistent and still name things that aren't there.
Comments