canary everything
Canary everything: full real installs as the test, not mocks
The most valuable infrastructure change I made this year wasn't a tool — it was a testing philosophy: the canary is a full, real install, run in a disposable environment, before anything touches production. No mocks, no "close enough," no "it worked in my head." If the real install doesn't come up clean on a throwaway node, it doesn't get to touch the one you actually use.
Why mocks and unit tests lie to you here
- Unit tests test your assumptions, not reality. They can't see that the real package needs a service you forgot, or that the network path is different.
- The bugs that hurt are integration bugs: the seams between components, not inside them.
- A green
make testis not the same as "this deploys."
What a real-install canary actually is
- A disposable node, provisioned from scratch (fresh VM, fresh install).
- The full, real install path — the same playbooks/flows that run in prod.
- A guard that refuses to touch a non-canary node (a hard safety boundary).
- A verify step that checks the thing actually works, not just that it installed (health probes, not exit codes).
- Tear down when done — the canary is disposable by design.
The two canaries that matter (concrete)
- K8s canary: install a fresh Argo node, join it to the real cluster as a non-routable member (so it can't steal traffic), run the post-install steps, verify the cluster is still healthy, then remove it. Proves the join/install path without risking the live members.
- Service canary: stand up the real service (e.g. the package mirror, the workflow engine) on a canary node with a canary-specific config, hit its health endpoint, compare against a baseline. Proves the service comes up and serves correctly before you promote it.
The details that make it safe
- Isolation by construction: the canary is a separate node with a separate identity — it can't be pointed at prod by accident.
- Health, not install: "the process started" is not a pass. The endpoint must answer correctly.
- Baseline + verify: capture what "good" looks like, then assert the canary matches. Drift is a failure, not a shrug.
- Idempotent teardown: a canary that can't be cleanly removed is a liability.
What it caught (the payoff)
- The whole point of the canaries is that they're where failures are supposed to happen. Every time one caught something before prod, it paid for itself.
- The pattern generalizes: any "install + configure + verify" flow is a canary candidate.
The principle
- You don't get to test your infrastructure on production.
- Build the throwaway, run the real install there, verify it works, then promote. If you can't afford the throwaway, you can't afford the change.