choosing a control plane
Choosing a control plane: why I run two, and what each one costs you
Every self-hosted setup has a control plane — the thing that decides what runs where, when, and how. The surprising part is that there's no single right answer; there's a trade-off, and I ended up running two of them, each for a different job. This is a map of that trade-off, and what each choice quietly costs you.
What a control plane actually does
- It's the source of truth for "what should be running."
- It reconciles desired state vs. actual state (or, in the imperative case, it just does a thing once and hopes).
- The quality of your control plane is the ceiling on how boring your operations can be.
The two poles: declarative vs. imperative
- Declarative (e.g. Flux/Kustomize on k8s): you declare the desired state; the system continuously reconciles toward it. Failure recovers by itself. Cost: a cluster to run, a steep learning curve, and a new single point of failure you have to engineer around.
- Imperative (e.g. Ansible over SSH): you run a playbook; it makes the machine match, then stops. Simple, no daemon, no cluster. Cost: nothing reconciles after it runs — drift is silent, and "did it actually apply?" is on you.
Why I run both (the split by job)
- The cluster (Flux/Argo): for the stateful, always-on services that need continuous reconciliation and a real deployment pipeline. The declarative model is worth the cost here because these are the things that must stay up.
- The bare nodes (Ansible): for provisioning, kernel pinning, and the one-shot setup that doesn't need a daemon. Running a full cluster just to manage a few VMs would be absurd.
- The line to draw: does this need to keep reconciling, or is it a one-time setup? Reconciling → cluster. One-time → Ansible.
What each one costs you (the honest ledger)
- Cluster cost: RAM for etcd + control plane, a TLS/PKI surface, a webhook and image-pull path, and the mental model of "the cluster is now a thing I operate."
- Ansible cost: no continuous truth — you must verify after every run, and drift is invisible until it bites you.
- Two-plane cost: two mental models, two auth surfaces, two places for secrets to live. The benefit (each tool doing what it's best at) has to be worth the seam between them.
The single-point-of-failure problem (and the fix)
- A control plane is, by definition, a SPOF. The job isn't to remove it — it's to make its failure non-catastrophic.
- The pattern: keep the control plane's outage from taking down what it controls. A Flux outage shouldn't kill the pods; an Ansible outage shouldn't kill the service.
- This is the whole reason for the canaries and the pinning: the control plane gets to be wrong or down, and the system still stands.
The principle
- Don't pick a control plane because it's fashionable. Pick it because the job (reconcile-always vs. set-once) fits the model.
- And in both cases: design so the control plane can fail without taking the system down with it.