War stories, build logs and debugging tales from a homelab that runs my whole apartment — two Proxmox nodes, ~24 services, and a cat named Pixel who is not allowed to trigger automations.
The listener that wakes my AI agent was running, connected and error-free for four days, and it had not received a single event. Every health signal was green because every health signal was about the connection, not the subscription.
Two false alarms one quiet morning led to a DNS fallback that every container listed and none had ever reached. It looked redundant for months because two nameservers were written down.
I made every box on my network map clickable, and before it shipped it sent me looking for a mystery device on switch port 8. Port 8 is my router. Two false alarms about the observer that is never in its own picture, and a TV port that dropped its link four times a day.
Two old laptop drives now hold a full, encrypted, add-only copy of my homelab: one plugged in, one in a drawer. Getting there crashed the backup VM once, ate a chunk store, and ended with a monitor that honestly said 'never' for 81 seconds.
The front page of michaelz.dev is a painting with a hooded guard, and the hood is the login. Now there's a starry rift in his face that opens into the live solar system, without breaking the sign-in and on a GPU budget written by a broken phone.
Two orchestrator self-checks, every five minutes and every six hours, had not run for 21 days, and systemctl said they were enabled the whole time. One Requires= line on a timer turned an ordinary restart into a silent off switch.
My main server has no remote console, so I armed a reboot watchdog to rescue a hung restart. During a kernel update it hung for six minutes and the watchdog did nothing. It never could have: it lives inside the kernel that had already gone.
A 4:52 alert said a person was in the flat while we were both away. It was the cat, and my own vision model had already said so. A cautious fallback sentence turned "I can't confirm" into "someone was detected". Plus: deciding which failure direction is the dangerous one.
Everything Was Green, part III. My camera narrator kept going blind for half an hour at a time. Four days of theories rested on a benchmark that was never a control. The cause was one frame in seventy, and a retry that faithfully re-sent the poison.
A new Diagnose button called a container I'd just used "unhealthy". Nine more agreed. The health check and the restart had different ideas of what each service is called, and the map itself named three programs that weren't there.
I asked my house assistant about a script and it answered fluently, with correct sources, about something else entirely — because my question contained the word "before". Auditing the neighbouring words found worse, including one hiding inside "keep-alive".
My camera's away-alert had been switched off since July, and every health check was green. Worse: the house briefing had been confidently reporting "all quiet" the whole time — because the only thing that writes that data was the thing that was off.
My camera's AI narrator started failing with a parsing error. Six health checks, six green lights, and a feature that plainly didn't work. The cause was three programs quietly fighting over one graphics card — and 70x slower output getting truncated mid-sentence.
A deterministic engine with no AI in it, an AI agent that watches, a HUD where I get to say no, and a folder of markdown that remembers everything. How they fit together — and why the interesting part is how they refuse to trust each other.
The internet vanished from the whole apartment, with everything on my side healthy. A WAN capture showed my requests leaving and nothing answering — while the ISP's own gateway was alive on the wire. The proof it was upstream came from the one workaround that didn't work.
Moving one Siemens dishwasher onto an IoT VLAN took two sessions and a fight with Home Connect's idea of "home". The isolation I built to protect the network turned around and blocked me from managing the very device it was isolating.
I built DNS failover, tested it, watched it "fail", panicked, and made a real mess — before discovering my own diagnostic tools had been lying the entire time. The DNS never actually broke.
I clicked "Update" on Home Assistant and the whole apartment went dumb — stuck NOT_RUNNING. The cause wasn't my config. A download server three time zones away was returning 522, and the twist is that rollback couldn't have saved me either.
Full signal bars, right network, strong connection — and 100% packet loss. My living-room TV was a WiFi zombie, and the culprit was a radar-avoidance rule baked into the 5 GHz band I never should have been sitting on.
My friends' streams kept buffering, and the remote bitrate cap I set did absolutely nothing. Because behind a reverse proxy, every client on earth looks like it's sitting on my couch — until you teach the app who the proxy is.
My media pool was filling up with nothing new. Every movie existed twice — once seeding, once in the library — and the fix was one line of filesystem theory I'd drawn a boundary straight through.
A sequel. The network was always there — my laptop just refused to roam properly. Three ghosts (an amnesiac daemon, a phantom radio, a port poltergeist), one watchdog, and the one bug I chose to live with.
Her name is Pixel, she's four kilograms, and she's the single most important constraint in my entire home-automation design. Why every scene in my apartment is a manual button.
A disk filled up, and by the time I reached for my phone it had already been investigated and reported. Building an autonomous ops brain — and the layers of paranoia that make it safe to run.
I run ~24 services and not one Docker container. Native systemd in unprivileged LXC instead — the reasoning, the real tradeoffs, and who should (and shouldn't) copy it.
Building the loop I actually wanted from a homelab: photos leave my phone, land in my own cloud, and play on the TV — without my 400 screenshots of parking signs tagging along.
All I wanted was a Google-style month view in Home Assistant with events that span days as bars. What I got was a multi-hour fight with a shadow DOM and the word "+more". Here's how it ended.
My backup node kept vanishing from the network at random. No panic, no logs that mattered — just silence. The culprit turned out to be a 15-year-old Intel driver bug, and the fix was three words long.