Choose your database foundation with a bake-off, not a debate

Part 3 of the scale-to-zero Postgres series.

The platform's foundation — self-hosted Neon storage — started as an inherited decision. The architecture doc chose it, the MVP shipped on it, and iteration after iteration hardened it. Then an independent architect review landed the sentence that reframed everything:

"The foundation was decided by inheritance, not evidence. You are paying the full ops cost of a disaggregated Rust storage cluster while none of the capabilities that justify it are actually in use."

Ouch. And: fair. The review demanded a bake-off promoted to a decision gate — stop hardening until the alternative is measured.

The contender, and the rule that made it fair

The alternative was the boring-excellent choice: CloudNativePG (the operator knext itself uses) with declarative hibernation — one operator, one pod, one PVC, against Neon's six-workload plane. The fairness rule: both foundations sit behind the byte-identical gateway binary. Wake-on-connect for CNPG is the same code path in exec mode, toggling the hibernation annotation. If the proxy differs, the comparison is theater.

Twenty samples per cell — cold wake, warm connect, reconnect-after-drain — plus failure drills and an honestly counted ops-mass inventory.

Round one: the case for switching

CNPG's 14.4s looked disqualifying — until instrumentation showed Postgres was ready in 24 milliseconds; the pod waited out a 10-second kubelet readiness probe tick. One declarative line (periodSeconds: 1) later: 6.3s, with an irreducible floor around 4–5s (pod re-creation is the cost hibernation can't dodge).

Now the honest math got uncomfortable. Neon's edge was ~1.7×, none of its differentiators (branching, PITR) were wired to anything, and the consumer's own stack defaulted to CNPG. The bake-off's written recommendation: switch to CNPG; keep Neon as an escalation lane. The evidence said so, and I'd have defended that call.

Round two: the measurement that flipped it

The owner deferred ratification and asked one sharp question: what are the floors? Two parallel experiments answered:

• CNPG's floor is the pod. Every wake re-creates one; ~4–5s is where tuning ends. There is no warm tier short of simply staying on.
• Neon's floor is the attach. Because compute is stateless, a pod can exist parked — booted but gated before Postgres starts. Waking is opening a gate: no scheduling, no container start. Measured: 413ms p50, 206ms best, at the honest price of a parked pod's 256Mi.

That asymmetry is structural, not tunable: separated storage makes a sub-second tier possible; hibernation architecture makes it impossible. And crucially, the warm tier answered the review's real objection — Neon finally had a differentiator that would be used, by the platform's headline behavior, not listed in a brochure. Ratified: Neon, two-tier — cold-zero default, warm opt-in — with CNPG preserved in the repo as the documented simplicity alternative and the bake-off harness kept runnable.

Kill criteria: the decision that can fire itself

The part I'm proudest of isn't the choice — it's that the choice ships with standing tripwires, written into the ADR: pivot off self-hosted Neon if storage ops exceed ~1 engineer-day/month, if the wake edge evaporates, if the version-pair treadmill costs more than an upgrade sprint per quarter, or if the capabilities justifying it go unused for a quarter. One later review formally ruled a criterion cleared (the storage SPOF, after automated failover shipped); the others stay armed. An architecture decision without falsifiers is just a preference with documentation.

What to steal

• Same-harness or it doesn't count. The byte-identical gateway is what made the numbers comparable — and proved the glue was foundation-agnostic, which itself lowered the cost of deciding either way.
• Instrument before you rank. Both foundations' headline numbers were Kubernetes artifacts (probe ticks, DNS caches), not engine truths. A bake-off that skips root-causing measures your YAML, not your options.
• Let the decision be wrong once. The first recommendation was reasonable and would have been a mistake. The extra evidence round cost two days and was the difference between a defensible decision and a correct one.
• Write the kill criteria while you still remember your doubts. Future-you will be too invested to invent them honestly.

Next: disaster recovery you can rehearse — including reverse-engineering a binary format to make restores writable.