The Ghost in the NIC: chasing an e1000e hardware hang

My backup node had a haunting. Every so often — no pattern I could see — it would simply fall off the network. The machine was still running. The services were still up. But it was unreachable, a ghost in my own rack, until I walked over and power-cycled it.

It took a while to catch, because everything about it screamed "it's fine." This is the story of cornering an intermittent hardware bug, which is a genre of debugging where the hardest part is just believing the hardware is the problem.

The symptom

The node — my secondary box, the one that runs backups and acts as a fallback — would go dark. SSH: timeout. Ping: nothing. But the other node, the management UI, every other device in the apartment: perfectly fine. So it wasn't the switch, wasn't DNS, wasn't the router. It was this one machine, and only sometimes.

Intermittent and isolated to one box is the worst combination. You can't reproduce it on demand, and by the time you notice, the evidence has usually rebooted away.

Following the only clue

The trick with a machine that loses its network but keeps running is that the kernel still knows what happened — it just can't tell you over the wire. So after a power-cycle, the first stop is the kernel log:

journalctl -k | grep "Hardware Unit Hang"

And there it was. Not once. Tens of thousands of times. Over a single bad stretch, the log had recorded more than twenty-six thousand of these:

e1000e ... Detected Hardware Unit Hang

The e1000e driver runs Intel's onboard gigabit NICs — the I218/I219 family you find on a lot of small-form-factor and used office machines. And it has a long, well-documented history of exactly this: under certain conditions the NIC's transmit logic wedges itself, the driver detects the hang, tries to reset, and sometimes never fully recovers. The hardware is, technically, working. It's just stuck.

The hardest moment in this kind of debugging is the mental shift from "what did I misconfigure?" to "the silicon and its driver are fighting each other." Everything in you wants it to be your fault, because then you can fix it cleanly.

The three-word fix

You can't patch the silicon, but you can take away the features that trigger the hang. The offloads — TCP segmentation offload, generic segmentation offload, generic receive offload — let the NIC do clever batching of packets. They also happen to be what tips this particular chip over the edge. Turn them off and the hang stops:

ethtool -K <nic> tso off gso off gro off

Yes, you give up a sliver of efficiency. On a home backup node moving data over gigabit, you will never notice. What you will notice is the node no longer disappearing.

Making it stick

A live ethtool command evaporates on reboot, so I wrapped it in a tiny systemd service that runs at boot and applies the settings before anything important comes up. One important detail learned the hard way: the NIC can hang during the reboot itself, before the mitigation service has a chance to run. So if that node ever goes dark right after a restart, the playbook is simple and slightly undignified: assume the ghost is back, and power-cycle it.

The lesson

Two things stuck with me. First: when a machine loses the network but keeps running, the kernel ring buffer is your best friend — the answer is almost always sitting in journalctl -k, waiting for you to come back and read it. Second: cheap, used, small-form-factor hardware is a fantastic way to build a homelab, but it comes with decades of accumulated driver folklore. Sometimes the fix isn't elegant. Sometimes it's three words and a shrug. And sometimes that's exactly enough.