the age key incident
The incident that taught me about key custody: losing an age key and rebuilding trust
I once had a system I couldn't recover from — not because it was down, but because the key to it was gone. It was an age secret key, the kind of thing that "just decrypts some files," and losing it felt small. It wasn't. This is the story of how a single lost key became a full incident, and the custody discipline I built so it can't happen again.
What actually happened
- A component (the thing that sealed/encrypted its state) depended on an age secret key to unseal itself.
- The key was treated as a detail — written down somewhere, maybe in a config, maybe in a comment, maybe "in my head."
- At some point the key and the thing it unlocked got out of sync: the state was sealed with one key, but the key I had was the wrong one (or lost). The system was up, but locked, and I didn't have the key that could open it.
Why a "small" key is actually a load-bearing wall
- An age key is a capability: whoever has it can unseal that state. That makes it as sensitive as a password, a token, or a private key — except it usually gets less care than those.
- The failure mode is lopsided: losing the state is bad; losing the key is worse, because the key is what makes the state meaningful.
- The worst case isn't "I lost a file." It's "I have a sealed box and no combination," on a system I can't afford to be down.
The custody failures (the real root cause)
- No single source of truth for where the key lives.
- No verification that the key I have actually opens what it's supposed to open (I found out the hard way, not in a test).
- Key and state managed independently — they should be one unit, with one owner, one lifecycle.
- Assumed recoverability — "I'll find the key if I need it," which is not a plan.
The discipline I built afterward
- Keys are first-class artifacts: named, versioned, stored in the secrets store (not scattered), and tested — I verify a key can actually unseal a known sample before I trust it.
- Key + state are bound: the thing that seals state and the key that opens it are managed together, so they can't drift apart.
- Recovery is rehearsed: I practice the "I lost the key" path on a disposable node (a canary) so the real recovery is a procedure, not a discovery.
- Rotation is possible: if I can't rotate a key, I don't really control it.
The broader lesson
- Every secret is a capability, and every capability is a trust relationship.
- The question to ask about any key isn't "where is it?" but "can I prove, right now, that it does what I think it does — and what do I do if the answer is no?"
- Treat the key with the same rigor as the system it protects. Because in the moment it matters, that's exactly how much it's worth.