A while back I got a notification that one of my nodes was running low on disk. By the time I picked up my phone to deal with it, a second notification had already arrived: the situation had been investigated, the biggest offenders identified, and a report filed. I hadn't done anything. Something else had.
That something is a small service I've been building that I think of as the lab's autonomous operations brain. It runs in its own container, it watches everything, and within carefully drawn lines, it's allowed to act. This is the story of what it is and — more importantly — how you build something like this without handing root access to a robot.
The problem it solves
A homelab is a tiny data center with a staff of one, who also has a day job and would like to sleep. Most "incidents" are boring and well-understood: a disk fills up, a backup job fails, a scrub hasn't run in too long, a service stops responding. Each one has a known first response — the exact set of commands a tired human would type after being woken up. That's the perfect thing to automate: not the judgment, the legwork.
How it thinks
The system has four moving parts, and the design is deliberately boring in the right places:
1. A map of everything
At its core is a knowledge graph of the whole lab — every node, container, service, storage pool, and the relationships between them. "This service runs in that container, which lives on that node, which owns that pool." When something breaks, the brain can reason about blast radius: if a node goes down, what exactly is affected, and in what order does it cascade.
2. Events come in
Alerts from the monitoring stack — the metrics alertmanager, the uptime monitor — get normalized into a common event shape and turned into incidents. A scheduler also fires synthetic events on a timer ("has a backup happened today?", "is any pool overdue for a scrub?"), so the brain is proactive, not just reactive.
3. Workflows respond
Each incident type maps to a workflow — an ordered set of steps. respond_disk_full SSHes in, runs the disk and dataset usage commands, finds the biggest consumers, and reports. respond_backup_failed checks the backup server's API and confirms what's actually missing. The workflow doesn't improvise; it does the legwork a human would, and surfaces the findings.
4. It's all served from one app
The whole thing is a single FastAPI application served by uvicorn — the API, the scheduler, and even the React web UI all live in the same process. The dashboard that shows live incidents and recent actions? It's just static files served by the same Python app that does the thinking. No separate web stack, one thing to run.
The part that matters: not trusting it
Here's the uncomfortable truth about giving software the ability to act on your infrastructure: the hard part isn't the automation, it's the restraint. An agent that can run commands on your nodes is a liability unless it is boxed in from every direction. So the brain is wrapped in layers of paranoia:
- Dry-run by default. New workflows don't touch anything until they're explicitly graduated to live, one at a time, after being watched.
- Per-workflow policy. Read-only investigations can run freely. Anything that changes state has to pass a policy check.
- Blast-radius guards. The highest-value containers (the database, the cache) are flagged as always-protected — actions touching them require extra confirmation.
- Approval gates. Genuinely destructive operations don't just happen. They request approval, with a timeout, and wait.
- Capabilities, not shell. The brain can't run arbitrary commands. It can only invoke a fixed, audited set of "capabilities" — and every action is written to a durable log.
The result is a system that's allowed to gather facts and fix the safe, boring things on its own, but has to ask before doing anything that could ruin my evening.
Autonomy is easy. Bounded autonomy — the kind you'd actually let near production — is most of the work.
What it feels like to live with
Honestly? It feels like having a junior SRE who never sleeps and never complains. The 3 a.m. disk alert still happens, but now it arrives pre-investigated. The backup-didn't-run check runs every morning whether I remember to think about it or not. And the dashboard gives me a single pane where I can see what the brain has noticed and what it's done about it.
It is wildly over-engineered for a two-node apartment lab. That is, of course, entirely the point. The lab isn't just where my services run — it's where I get to build the kind of system I'd never be allowed to build at work without a committee. And occasionally, it fixes a problem before I even know I had one.
Comments