◂ writeups // michaelz.dev

Bricked by Someone Else's Outage: when a CDN wouldn't let my house boot

I did the most boring thing you can do to a homelab: I clicked "Update" on Home Assistant. A point release, the kind that lands every month. Ten minutes later the lights were still lights, but nothing in the apartment was smart anymore — no automations, no dashboards, no app. Home Assistant was stuck, and the reason had nothing to do with anything on my server. A download server three time zones away was having a bad day, and it took my house with it.

The update that never came back

The upgrade was routine: Core 2026.6.4 up to 2026.7.0. It downloaded, it restarted, and then it just... didn't finish. The supervisor reported Core as NOT_RUNNING. Not crashed, not erroring in a loud way — hung, mid-boot, forever. Every couple of minutes it would try again and land in the same place.

My first instinct was that I'd broken my own config. That's usually the honest bet. So I went to the one place that actually tells you the truth: ha core logs. And there it was, repeating:

Unable to install package X==Y ... wheels.home-assistant.io ... HTTP 522
...
Bootstrap stage 1 timeout

Two lines that reframed the entire problem. This wasn't my YAML. Home Assistant was trying to download something at boot, failing, and timing out its own startup because of it.

Why a running system phones home to boot

Here's the piece I didn't fully appreciate until it bit me. When Home Assistant Core changes versions, it re-fetches the Python "wheels" — precompiled dependency packages — for its integrations. On a normal Linux box those would already be cached on disk. But on Home Assistant OS, the deps live inside the Core container, which is ephemeral: a version change effectively gives you a fresh container. So every core version bump re-installs those wheels from scratch, and it gets them from one place: wheels.home-assistant.io, Home Assistant's own package CDN.

My smart home's ability to boot was quietly dependent on a web server I don't own being up at the exact moment I clicked "Update."

That morning, the CDN was returning HTTP 522 — a "connection timed out" from the edge, meaning the origin behind it was unreachable. And Home Assistant does not fail fast on this. Each failing wheel install retries, roughly 90 seconds a go, three times over. Stack up a few integrations that need a fresh wheel and you blow straight past the "Bootstrap stage 1 timeout" budget. Core never reaches RUNNING, so it never serves the dashboard, so all you see from the outside is a house that went dark after an update.

The trap: rollback doesn't save you

The obvious move when an update bricks something is to roll back. Home Assistant even keeps backups for exactly this. So I reached for it — and then stopped, because I realized it would walk me straight into the same wall.

Restoring a backup, or downgrading, is also a version change. It also rebuilds the Core container. Which means it also needs to pull wheels from the very CDN that's down. The escape hatch runs through the same locked door. This is the part that makes a CDN outage genuinely nasty: the standard recovery path has the same dependency as the thing that broke.

And I couldn't just install the packages myself, either. The HAOS SSH add-on is deliberately sandboxed — no docker, no pip, no python3. You cannot reach into the Core container and hand it the wheels by hand. Whatever the fix was, it had to be about getting Home Assistant to boot without those downloads.

Confirming it was them, not me

Before doing anything drastic, one command to be certain the CDN really was the villain — poke it directly and read the status code:

curl -m6 -o /dev/null -w '%{http_code}' \
  https://wheels.home-assistant.io/musllinux-index/<pkg>/

It came back 522. Not a network problem on my end, not DNS, not my config — the origin was simply down. Good: that meant the fix was to get Core to boot degraded, skipping every integration that needed a fresh wheel, and wait the outage out.

Booting the house on one leg

The recovery is unglamorous but reliable: temporarily remove every reason Core has to hit the CDN, let it come up, and put it all back once the CDN returns. Concretely, three kinds of offenders had to be quieted:

Custom integrations — the ones from HACS — just get moved out of the way so the loader skips them:

mv /config/custom_components/<x> /config/<x>.disabled

Config-entry integrations get disabled by editing their entry in the storage file, flipping them off so Core doesn't try to set them up:

// in .storage/core.config_entries
"disabled_by": "user"

And the sneaky one: deprecated YAML that still loads a component needing a wheel. Even a platform Home Assistant now rejects will still pull in that component's requirement before rejecting it — so a legacy block you thought was harmless can be what's dragging in the failing download. It has to come out of configuration.yaml too.

With all three quieted, ha core restart — and this time Core reached RUNNING. The house was smart again, minus a handful of integrations, which is a perfectly livable state for a few hours. When the CDN came back with a clean 200, I reversed every change (mv the folders back, set disabled_by to null, restore the YAML), restarted once more, and everything returned.

One habit that saved me a false victory: don't trust the restart's exit code. Verify the actual state over the API — /api/config should report "state": "RUNNING". A successful restart command and a running Home Assistant are two different claims.

The footnote that made it worse

2026.7.0 also happened to remove the old YAML ping binary-sensor platform. Fine on its own — it auto-migrates to a config entry. But the ping component needs icmplib, which is a wheel. So during the outage it was one more thing quietly reaching for the dead CDN, and one more thing I had to strip out of the config to get a clean boot. A breaking change and an outage, colliding on the same day, is exactly the kind of luck this hobby specializes in.

The lesson

Two takeaways I'm keeping. First, and uncomfortably: "self-hosted" does not mean "self-contained." A platform can run entirely on my hardware and still have a hard boot-time dependency on someone else's uptime. The failure didn't announce itself as "external CDN down" — it announced itself as "your update bricked your house," and the distance between those two descriptions is where you lose an hour.

Second: when the recovery path shares a dependency with the failure, you don't have a recovery path. Rollback felt safe right up until I noticed it needed the same dead server. The move that actually worked was orthogonal to the outage entirely — boot degraded, skip the parts that phone home, and wait. Sometimes the fix isn't undoing the change; it's learning to run without the thing that's missing until it comes back.

Comments

◂ all writeups michaelz.dev ▸