There is a camera in my hallway that describes what it sees, in a sentence, in plain English, on a machine in my own flat. One evening it stopped and started saying "could not analyse the frame — the model did not return the expected format". Every single health check I had said the system was fine. They were all telling the truth. The bug was real, serious, and filed under a department I wasn't looking in.
What it does, and why it's local
The idea is small. When the camera notices a person, a vision model looks at one frame and writes a short description — "a person in a dark jacket by the door" — which lands on my dashboard, and buzzes my phone only if the flat is empty. It's the difference between an alert that says something moved and one that says what moved.
It runs entirely on hardware I own. No frames leave the building. That's the whole point, and it's also the constraint that eventually caused this: running your own models means you own the resource problem too.
Every instrument said fine
Here's what I checked when it started failing, and what each thing told me:
- The service — running.
- Its health endpoint — returning OK, with the right model name.
- My uptime monitor — green, as it had been all week.
- The graphics card's own status tool — healthy, sensible memory figures.
- The service's log — not one error. Not a warning. Nothing.
Six instruments, six green lights, and a feature that plainly did not work. This is a specific and disorienting kind of stuck: you can't even begin bisecting, because nothing is pointing anywhere.
An error message names the symptom. The symptom is often in a different department than the cause.
The one tool that told the truth
The model server has a command that lists what's currently loaded. I ran it mostly out of habit, and there it was, in a column I had never once needed to read:
34% / 66% CPU / GPU
The model wasn't running on the graphics card. A third of it was running on the processor. Nobody had told me because, as far as every other tool was concerned, nothing was wrong: the service was up, the card was healthy, the model was loaded. It was simply loaded in the worst possible way.
Three tenants, one card
Over a few months, three separate things had quietly ended up sharing one consumer graphics card: a general-purpose assistant model, a small text model that writes my daily house briefing, and an embedding model that powers search over my notes. Between them they had a standing claim on most of the card, held open deliberately so none of them would be slow.
Then I added the camera's vision model — the newcomer, and the one with the least room left. It needed roughly four and a half gigabytes free to commit to loading fully onto the card. What was left was just under that. So it did what these systems do: it took what it could get and put the remainder on the CPU.
I measured it both ways, because I wanted the number and not the vibe:
- With plenty of room free — fully on the card, 1.3 seconds per frame.
- With the card nearly full — split onto the CPU, 97.5 seconds.
Seventy times slower. Same model, same code, same image. The only variable was how much room was free at the moment it loaded.
Why a memory problem produced a parsing error
This is the part I find genuinely delightful, in the way you can only find something delightful after you've fixed it.
When the model generates an answer, it's capped at a certain number of words — a sensible guard, so a rambling model can't run forever. The answer is supposed to come back as a small structured object. When the model was running at full speed, it comfortably finished inside the cap. When it was crawling at a third CPU speed, the cap cut it off mid-object — a fragment of structure with no closing bracket.
My code did exactly the right thing with a truncated fragment: it failed to parse it and reported, honestly, "the model did not return the expected format." Which is true! It's just three steps downstream of the actual problem. A memory-placement bug had put on a parsing bug's clothes and walked right past me.
The worst case was the case that mattered
Then I worked out when this happened, and it was almost comically badly timed.
The small model that writes my house briefing loads on demand — specifically, when I arrive home. So the sequence is: I come home. The briefing model loads and takes its share of the card. Moments later the camera sees me and the vision model tries to load into what's left.
In other words, the one moment the system was guaranteed to be degraded was arrival — which is exactly and only when the away-alert matters. I reproduced it deliberately three times to be sure. Ninety-seven seconds, every time.
The fix, and the decision I reversed
Two days earlier I'd made a call about this same card that felt obviously right: the newest arrival should be the polite one, so I'd given the vision model a short lease and let it release its memory quickly when idle.
That was the wrong axis. Politeness isn't the thing to optimise here — latency-criticality is. The vision model serves events: nobody presses a button and waits for it, and an alert about someone at your door is worthless if it arrives a minute and a half later. The other models serve requests, where a few seconds of warm-up is invisible.
So I inverted it. The vision model now holds its place on the card permanently, and the briefing model gives up its standing claim and reloads on demand. Which means the briefing model is now sometimes the one that gets split onto the CPU — and that's fine, because a small text model degrades gracefully. It went from instant to about five seconds. A vision model does not degrade gracefully; it goes off a cliff.
I verified it on the exact sequence that had been failing rather than in isolation: vision model claims its slot, briefing fires and takes the split, camera runs again immediately afterwards. 2.9 seconds, fully on the card. Through the full path my dashboard actually uses: 1.3 seconds.
What I'd take from this
Three things, and the last one is the one I keep relearning.
Shared hardware needs a stated priority. The moment more than one thing wants a finite resource, "they'll work it out" is not a policy — it's a race, and races are won by whoever happened to start first. Deciding which tenant is allowed to be slow is a design decision, and if you don't make it explicitly it gets made for you, differently, every time something restarts.
Steady-state size doesn't predict fit. The model settles at about 2.9 GB once it's loaded, which made "there's four gigs free, it'll fit" sound completely reasonable. It needs considerably more headroom to decide to load fully than it occupies afterwards. The number you can look up is not the number that governs the decision.
Health checks answer the question they were given. Every one of my six green lights was accurate. "Is the process running?" — yes. "Does the endpoint respond?" — yes. Not one of them asked "is it fast enough to be useful?", because I never thought to ask that, because it had never been slow. The gap between working and working usefully is where this class of bug lives, and no amount of green tells you anything about it.
The instrument that broke the case was one I'd never had a reason to run, showing a column I'd never had a reason to read. That's not a satisfying lesson — "check the thing you didn't think of" is useless advice. The useful version is narrower: when every instrument says fine and the thing plainly isn't, stop asking your instruments whether it works and start asking something, anything, how fast.
Part II is about the opposite failure, the same evening: a monitor that was green because it had been switched off for six weeks.
Comments