This week I gave an AI agent permission to restart a service in my apartment without asking me first. Exactly one service. One line of configuration. Getting to the point where I was comfortable with that single line took four separate systems and about six weeks — and the four systems are mostly interesting for the ways they refuse to trust each other.
I wrote about the first of them a while back, when it was one service that investigated disk alerts on its own. It's grown. What follows is the whole shape of the thing: what each piece is, why it exists, and where the seams are.
Four parts, and the division of labour is the entire design:
- The orchestrator — the hands and the rules. Knows every machine and what it's allowed to do to them. No AI inside it whatsoever.
- Argus — an AI agent that watches the lab and investigates. It uses the orchestrator, the same way I do.
- The HUD — the screen where I see what both are doing, and approve or veto it.
- The vault — a folder of markdown that all three treat as memory.
The important idea, and the one I'd defend hardest: the AI is never the safety layer. The guards are deterministic code that behaves identically whether the caller is me, a cron job, or a language model.
1. The orchestrator — the part with no AI in it
About fifteen thousand lines of extremely boring Python behind an HTTP API, running in its own container. It's boring on purpose. Everything that decides whether an action is permitted lives here, and none of it involves a model.
It starts with a map
The foundation is a typed graph of the lab — 82 entities and their relationships. Nodes, containers, VMs, services, storage pools, and typed edges between them: runs on, depends on, backed up to, hosted in. That graph is what makes this more than a script runner. Before touching anything, it can compute blast radius: if this breaks, what else breaks, and in what order.
It's also what lets me ask hypotheticals. "What happens if I reboot the primary node?" gets answered from the graph, without touching a single machine.
Tools and workflows
On top of the graph sit 25 tools — atomic operations like "get container status", "check pool health", "send a notification" — each declaring whether it reads or writes. Above those, 18 workflows: ordered procedures built from tools. Respond to a stopped container. Respond to a full disk. Report on certificate expiry.
Workflows are what actually run. Tools are what they're made of. The agent can't invent a new capability; it can only call what exists.
The guard stack
Every write passes through all of these, in order:
- Capability boundary — the caller's permissions must cover the tool.
- Execution policy — a write tool with no policy defaults to dry-run. Safe by default, not open by default.
- Blast-radius blocks — the reverse proxy, the auth server, the database and the cache are permanently forced to simulation. So are both physical nodes.
- Graduation — trust is earned per (caller, workflow, target) triple, walking from dry-run to observation to graduated. Not graduated means forced dry-run.
- Breakers — repeated failures automatically roll a pair back.
- Rate caps and cooldowns, plus a lock so only one workflow touches an entity at a time.
- Preflight re-check — it re-verifies the problem still exists before acting, so it never "fixes" something that already self-healed.
- Approval gate — live writes need me. Tokens are single-use and expire in five minutes.
- A global kill switch, re-read at dispatch so a stale copy in memory can't keep acting.
Closed by default, opening one specific, earned, revocable step at a time.
2. Argus — the agent that watches
Argus is a headless AI process on my laptop that watches the lab and investigates things. It was called Sandman until recently; I renamed it when I realised I'd named my always-watching monitor after the mythological figure who puts you to sleep. Argus is the one with a hundred eyes, who never closes them all at once.
It wakes up two ways, and the difference matters more than it sounds.
A weekly patrol runs on a timer, sweeps the fleet read-only, writes a report into the vault, and only pings my phone if something actually needs me. It checks its own budget first — if I've burned most of my model usage that week, it skips entirely rather than eating a session on a quiet Monday.
An event listener runs continuously. And here's the design decision I like most in the whole project: the always-on part is not the AI. The listener is a small, dumb, deterministic Python client on the orchestrator's event stream. It filters for genuinely critical events, deduplicates, enforces a cooldown and an hourly cap — and only then does it wake a read-only AI to go investigate.
The cheap part runs forever. The expensive part runs when something real happened. A language model in a while true loop is a bad architecture and an even worse bill.
3. The HUD — where I get to say no
If a system asks for approval and I don't notice, the approval gate is decorative. So the HUD's job is mostly to be impossible to ignore.
It's vanilla HTML, CSS and JavaScript — no framework, no build step, three static files. It has views for the overall state, the live topology graph, pending approvals with their blast radius and a countdown, the agent's patrol reports, and what's currently allowed to act alone.
A pending approval signals in three places at once: a stat tile turns yellow, the navigation grows a badge, and the icon rail gets a dot. Whichever view I'm on, I can't miss one. That's not polish, it's the feature — an approval gate is exactly as good as its worst notification path.
4. The vault — memory that outlives all of it
Underneath everything is a folder of about 220 markdown files in git. Architecture pages, one page per machine, one per service, incident logs, runbooks, project plans. Every hard-won lesson from three years of breaking things lives there in plain text.
Plain markdown is a deliberate choice. If every piece of software described in this post stopped running tomorrow, the knowledge is still readable in any text editor, still greppable, still in version control. Nothing about my memory depends on my infrastructure being up.
But grep is a bad way to remember things
Finding something in 220 files meant keyword guesswork across ten folders and some very long pages. So the vault got a retrieval layer: a private RAG running entirely on my laptop's otherwise-idle GPU.
It indexes the vault plus my other documentation into a local vector store — currently around 3,400 chunks — and answers questions with citations. A local embedding model turns the text into vectors; a 7-billion-parameter model does the answering. Nothing leaves the machine. Some of those pages describe my network in detail, and that was non-negotiable.
Two design notes that were worth the trouble:
Retrieval is hybrid. Pure semantic search is bad at exact tokens — an IP address, a kernel flag, a driver name. So every query runs semantic and keyword search in parallel and fuses the rankings. Embeddings find "the thing about the flaky network card"; keyword search finds e1000e.
Secrets are excluded at index time, not query time. There's an opt-in allowlist of folders, a denylist of filenames, and a scanner that drops any chunk containing actual key material. A secret that was never indexed cannot be retrieved by any interface, by any caller, through any bug. Filtering at the query layer would mean the data is in there and I'm relying on the filter — a much worse position.
The mistake that taught me something
I added a memory store on the side, so I could correct the thing: "actually the access point is on channel 44 now, not 100." Corrections override the documents when they conflict. It worked.
Except only for the path where the local model writes the answer. The search path returns raw document chunks — it never saw my corrections at all. So the agent, which uses search, was reading documents that still claimed my network was flat after I'd spent two weeks putting VLANs into it.
The fix wasn't clever, it was procedural: a promote loop that lists every correction not yet reflected in a real document, and shows each one next to the page it most likely belongs in, so the edit lands in the right place. Corrections are where a lesson is captured. The vault is where it becomes true. Promotion is the step between, and it needs to be visible or it doesn't happen.
The missing sense: what changed?
For a long time the system knew what had broken (incidents), what it had done (audit log), and what the documentation said. It had no idea what had changed. And almost every incident in this lab follows a change — that correlation lived entirely in my head.
So now it takes periodic snapshots over its existing connections — installed packages in every container, enabled services, guest configuration, router and access point settings — and diffs them. No agents installed anywhere. Only the newest snapshot is kept; the diffs are the permanent record, so it never grows.
The query that makes it worth having is "what changed near this thing, around when it broke?" — and the important word is near. It expands through the graph before searching, because the change that broke a service is almost never recorded against the service. It's recorded against the container it runs in, or the node underneath that. Asking about a media server also searches its container and its host. That graph expansion is the difference between a diagnostic aid and a changelog.
Argus gets that as a tool. It can ask what changed before it forms a theory, which is roughly the first thing a competent human does and roughly the last thing an eager one does.
What it's actually allowed to do alone
Which brings me back to that one line of configuration.
Until this week there was no autonomous write path at all. Graduation and approval were independent layers, so even a fully-graduated action still stopped and asked me. Adding a path meant answering a question I'd been avoiding: which things, specifically?
92 pairs were already graduated. If graduation alone had conferred autonomy, shipping the feature would have made 92 things autonomous in a single commit — which is exactly the kind of change that looks fine in review and terrible at 3 a.m. So autonomy requires being on a separate, explicit allowlist and being graduated. Trust is granted one triple at a time.
Today that allowlist has one entry: the agent may restart my uptime monitor. That's it. If it can't handle the most boring service in the building, it doesn't get a second one.
The breaker nobody builds
The existing breaker counted failures. But once writes are autonomous, the failure mode that actually bites is different: a restart that succeeds every single time while the service keeps falling over. Every run reports success, so a failure-counting breaker never trips, and the rate limit just paces the loop indefinitely. You get a perfectly healthy-looking automation quietly babysitting a broken thing forever.
So there's a second breaker that counts remediations. Three autonomous fixes of the same thing within six hours means the fix isn't holding — freeze it and escalate to a human. It writes that decision to disk before acting, so a restart can't erase the evidence of its own restart loop. And a frozen action stays frozen until I clear it, because automatic recovery would just re-enter the loop it was built to escape.
Granting is hard, revoking is easy
The last piece is deliberately lopsided. The file that grants autonomy is root-owned on a read-only mount, so the service literally cannot write to it. The file that revokes it is in the service's own state directory, and the pause button is on the HUD's front page.
The orchestrator cannot grant itself autonomy. It can always be stopped instantly. A kill switch that can be blocked by a filesystem permission is not a kill switch.
Was any of this necessary?
Absolutely not. This is a two-node lab in an apartment. A cron job and a monitoring dashboard would cover ninety percent of it, and I've said as much to people who asked whether they should build the same thing.
But the lab isn't only where my services run — it's where I get to build the kind of system that would need a committee and two quarters at work. And the constraint that made it interesting wasn't "make it autonomous". Autonomy is easy; you just remove the checks. The interesting constraint was making something I'd genuinely trust near things I care about, which turned out to be almost entirely a question of what it's forbidden to do.
Four systems, dozens of guards, a knowledge graph, and a local language model — so that one AI agent may, under five simultaneous conditions, restart one uptime monitor without waking me up.
I think that ratio is correct.
Comments