2026-10-18

pinning the kernel

When the kernel is the bug: pinning, booting the old one, and not upgrading at 2am

Most of my infrastructure is pinned to a specific kernel version — not because I'm afraid of new kernels, but because I learned the hard way that a kernel upgrade is the one change that can take down everything at once, including the tools you'd use to fix it. This is the story of why "don't upgrade the kernel at 2am" became a rule, and the machinery I built to enforce it.

Why the kernel is different from every other package

• A bad app update is a bad app. A bad kernel update is no network, no SSH, no way to roll back — because the thing that would roll it back is dead.
• The kernel sits under everything: drivers, networking, the filesystem cache. A regression there doesn't fail one service; it fails the node.
• And unlike an app, you can't just restart a kernel. You have to reboot — and a reboot with a broken kernel can brick the box.

The incident that set the rule

• A kernel update landed, and the next boot (or a driver change) broke something fundamental — the kind of failure where the node is up but unusable, or down entirely.
• The lesson wasn't "that kernel was bad." It was: I let a kernel upgrade happen at a moment when I couldn't safely test it. That's the actual failure.

The fix: pin, don't upgrade

• Pin the kernel version explicitly, in config, so a distro upgrade can't silently swap it out.
• Boot the pinned kernel — if it's not the default, make it the default, and verify the box actually came up on it.
• Isolate kernel changes from app changes: a kernel bump is its own deployment, with its own canary, its own rollback.
• Reboot deliberately, not accidentally. A reboot is a deploy; treat it like one.

The machinery that enforces it

• A role that installs the pinned kernel, sets it as the boot default, and verifies the running kernel matches the pin (not just that the package is installed — that the box is actually running it).
• The canary runs the kernel pin on a throwaway node first: does it boot? Does it come up clean? Only then does prod get the pin.
• The guard refuses to let a kernel change run on a non-canary node without the canary having passed.

The broader principle: "the thing that can't be rolled back is the thing

you test most"

• The riskiest change is the one with the worst rollback story.
• So it gets the most protection: the strictest canary, the most explicit verification, the most deliberate timing.
• A kernel isn't "just a package." It's the ground everything stands on — and the ground can't be rolled back with a git revert.

The rule, stated plainly

• Never let a kernel upgrade happen untested, unsupervised, or at a moment when you can't afford the node to be down.
• Pin it. Boot it. Verify it's the one running. Reboot on purpose.