package mirror generations
The package mirror is a system too: immutable generations and the mirror you can't trust
A local package mirror sounds like the safest thing in your homelab — it's just files, right? It's not. A mirror is a system with a trust model, a versioning model, and a failure mode that only shows up when you most need it to be reliable. This is how I came to treat the mirror as a first-class component: immutable generations, signed or at least verified content, and a canary that proves it actually serves what you think it serves.
Why a mirror is harder than it looks
- A mirror is a trust anchor. Every node that installs from it is trusting the mirror to serve the right bytes.
- It has a lifecycle: content is added, updated, and sometimes wrongly changed. A stale or corrupted file is a silent failure.
- It's on the critical path: if the mirror is down or wrong, nothing can be installed. That's a SPOF wearing a friendly face.
The failure modes that actually bite
- Silent corruption: a file that's truncated or half-written, and the package manager either fails obscurely or (worse) installs something broken.
- Drift: the mirror's content diverges from what you intended to serve — a partial sync, a race, a bad import.
- The "it's just files" trap: because there's no daemon to watch it, nobody notices it's wrong until an install fails at the worst time.
The fix: immutable generations
- Treat the mirror's content as generations: a set of files that were produced together, treated as atomic, and never mutated in place.
- A new generation is built in full, verified, then promoted — the old one stays available until the new one is proven good.
- This gives you a clean rollback: point back at the previous generation instead of trying to "fix" the current one.
Verification: the mirror must prove itself
- Hashes / signatures: content is verified against a manifest, not trusted because it's "in the directory."
- A canary that installs from the mirror: the real test of a mirror is a real install. Stand up a canary node, install a known package from the mirror, and check it works. If the canary can't install, the mirror is broken — full stop.
- Baseline + verify: capture the expected set of packages/versions, and assert the mirror serves exactly that. Drift is a failure.
The trust model, made explicit
- Who can write to the mirror? (Answer: as few people/automations as possible.)
- How do you know what's on it is what you meant to put there? (Manifests, not faith.)
- What happens when it's wrong? (Roll back to the last good generation — not a 3am debugging session.)
The principle
- A package mirror is not storage; it's a trust and versioning system that happens to be implemented with files.
- Give it the same rigor as the services that depend on it: immutable generations, verified content, and a canary that proves it works before anything trusts it.