In The four systems that run my house I described the design decision I liked most: the always-on part of my AI agent is not the AI. It's a small, dumb listener on the orchestrator's event stream that wakes a model only when something real happens. For four days in late July that listener was running, connected, error-free, and had not received a single event.
Everything I would have checked
I moved the listener onto its own container on a Wednesday evening and checked it. Then, four days later, I went to tick off the last box of that project: inject a fake "container stopped" event and watch the agent wake up and investigate it.
The orchestrator did its part perfectly. It took the event in, ran the right workflow, opened an incident with the right blast radius, and broadcast it. The listener logged nothing at all. That mattered, because the listener logs every event it drops, with the reason. Silence meant the event had never arrived.
So I looked at every health signal I had, and all of them said fine:
- systemd: active, zero restarts, the same process since the evening I moved it.
- The socket to the orchestrator: established.
- The listener's own log: not one "connection down" line in four days.
- Failed units on the box: none.
Every one of those was true. None of them was about the thing that had broken.
Asking a second client
There's no amount of staring at a four-day-old client that tells you whether the server still thinks it's subscribed. So I stopped staring and asked a question with two answers: I opened a fresh connection from the same container and fired another event.
The new client got it. The old one, same host, same broadcast, same second, got nothing. That one comparison moved the fault from "somewhere" to "the server has forgotten the old client, and the old client doesn't know".
A connection being up tells you about the pipe. It tells you nothing about whether anyone at the other end is still sending into it.
Evicted after ninety seconds
The orchestrator tidies up idle WebSocket sessions. Every 30 seconds it sweeps out any session that hasn't been "touched" in 90. That's sensible housekeeping. The problem was what counted as a touch: exactly two things, the server successfully sending to the client, or the server receiving a message from it.
My listener only ever listens. On a quiet fleet nothing happens, so nothing was sent to it, and it never said anything back. Ninety seconds after it connected, the server evicted its session and stopped broadcasting to it.
And eviction didn't close the socket. The server simply dropped the session from its list and left the TCP connection open. From the client's side nothing had happened: no close frame, no error, no reason to reconnect. It sat there, connected to a server that no longer knew it existed.
The keepalive that made it worse
The listener did send keepalive pings every 20 seconds. They didn't help, and in a way they hid the problem. Those are WebSocket protocol pings, answered by the web server layer underneath the app. They never reach the application code that counts as a touch. So they did keep the transport alive, which is precisely what made everything look healthy, while the subscription they were supposed to be protecting was already gone.
Why my own test missed it
I had tested the wake path when I set this up, twice, and both passed. Both events were fired within 45 seconds of a restart, inside the 90-second window before eviction. They proved the chain existed. They could never have shown that it decays when it sits idle, because they never let it sit idle.
The good news is the part that wasn't affected. Detection never depended on this listener: the uptime monitor, phone alerts, Home Assistant and the orchestrator's own self-healing all run without a model. What was lost for four days was the investigation, not the alarm.
The fix, from both ends
The funny part is that the server already had the answer. Right after the "touch this session" line sat a branch that ignored any incoming message saying ping. It existed for exactly this case, and nothing had ever used it. The listener now sends an application-level ping text frame every 30 seconds, which touches the session and keeps it subscribed.
And the server no longer evicts silently. The cleanup now returns the sessions it removed instead of a count, and closes each of their sockets properly. If a client is ever evicted again, it sees the close, and the reconnect-with-backoff it already had takes over.
To prove it, I did the two things the original tests hadn't. The new tests fail against the old code, 5 of 6 and 3 of 3, so they can actually catch this. And live, I waited 237 seconds, more than two and a half eviction windows, before injecting an event. The listener collected it, woke the agent, and the agent's report was committed.
What I'd take from this
- A healthy transport is not a live subscription. Active, established, zero restarts and a clean log are all statements about the pipe. None of them can see whether the other end still sends into it.
- Diagnose subscriptions with a second client. Same host, same event: it either gets the frame or it doesn't. That beats any amount of inspecting the first one.
- Eviction has to be loud. Removing a client without closing its connection creates a client that can never know it should reconnect.
- A test that only runs on a freshly started process can't see a decay bug. Let it sit idle past every timeout you know about, then assert.
Comments