Scale-to-zero Postgres on Kubernetes: a database that sleeps

This is Part 1 of a five-part series: 2 — the wake-on-connect gateway · 3 — the foundation bake-off · 4 — rehearsed disaster recovery · 5 — the 400ms warm tier.

My Next.js apps scale to zero on Knative. Their databases didn't — every idle side project kept a Postgres pod burning RAM around the clock. So I built the missing half: scale-zero-pg, a platform where each database consumes zero compute while idle and wakes on the first client connection.

The headline numbers, all drill-measured on a real OKE cluster:

Don't build the database — build the axle

The whole design rests on one reuse decision. Neon open-sourced (Apache-2.0) exactly the thing that makes this possible: a Postgres where compute and storage are separated. Their safekeepers hold a quorum-replicated WAL, their pageserver serves pages on demand, and the Postgres process itself becomes stateless — kill it, restart it, and it lazily fetches whatever pages queries touch. No volume. No restore. Cold start is independent of database size.

That means the only thing missing for scale-to-zero is glue:

client ──pg wire──▶ GATEWAY (Go, always on, stateless)
                      │  parse startup ▸ asleep? ▸ scale 0→1
                      │  ▸ hold the connection ▸ replay ▸ pipe
                      ▼
                    COMPUTE (Deployment, replicas 0↔1)
                    stateless Postgres 17 + neon extension
                      │ WAL out            ▲ GetPage@LSN
                      ▼                    │
                    STORAGE (StatefulSets, never scale to zero)
                    safekeepers ×3 · pageserver · object storage

The gateway is ~2,000 lines of Go. It speaks just enough Postgres wire protocol to read the startup packet, wakes the compute through the Kubernetes API, holds the client's connection while the database boots, then gets out of the way and pipes bytes. When the last connection closes and stays closed, it scales the compute back to zero. The app never knows.

Every latency villain was Kubernetes, not the database

This is the part I didn't expect. The database engine was never the bottleneck — Kubernetes mechanics were, three separate times.

Villain #1: CoreDNS negative caching. My first cold wakes took 5.2 seconds and no amount of pod tuning moved the number. The compute sat behind a headless Service, which has no DNS record while at zero replicas — so the gateway's first lookup got an NXDOMAIN that CoreDNS cached for ~5 seconds, masking every improvement behind it. Switching to a ClusterIP Service (whose record always exists) instantly halved the wake to 2.4s.

Villain #2: kubelet probe cadence. I bake-off'd the design against CloudNativePG with hibernation — its un-hibernate took a suspiciously consistent 14.4 seconds. Postgres itself was ready in 24 milliseconds; the pod sat invisible for a full 10-second readiness-probe tick before the Service would route to it. Setting periodSeconds: 1 cut the wake to 6.3s. The engine was never slow. The polling was slow.

Villain #3: the walreceiver kick. During disaster-recovery work we found the pageserver only starts streaming WAL for a timeline after a compute "kicks" its safekeeper. A restore helper that patiently waited for catch-up would wait forever — it worked in one test only because a crash-looping pod happened to be providing the kicks. Distributed systems teach humility in weird ways.

For the record, Neon's own share of a cold wake — attach, basebackup, Postgres start — is 123–160ms. Everything else is pod scheduling, container startup, and the two villains above.

The warm tier: sub-second, honestly priced

2.5s is fine for a hobby app waking from overnight sleep. It's not fine for everything. So there's an opt-in second tier: a gated pod that already exists but blocks before starting Postgres, polling a TCP "gate" port on the gateway (via bash /dev/tcp — the stock compute image has no curl). Waking is just opening the gate: no scheduling, no container start, only the ~150ms attach. Measured: 413ms p50.

The honest price: a parked pod reserves 256Mi around the clock. That's not scale-to-zero — it's a warm-RAM tier, and the docs say so in exactly those words. Because both computes attach to the same timeline, the gateway enforces the single-writer invariant in-band: the gate only opens after the Kubernetes API confirms the cold deployment is fully drained. The negative test — gate refuses while a cold compute exists — is the most important test in the repo.

Reverse-engineering my way to writable restores

Backups mirror the storage bucket off-cluster. Restores worked — but only read-only. Fresh safekeepers report flush_lsn 0/0, Postgres refuses to start read-write, and Neon's OSS release ships no API to create a safekeeper timeline at an arbitrary LSN. The docs said: manual gap.

The fix went under the API: reverse-engineer safekeeper.control (magic 0xcafeceef, version 9, crc32c trailer), write a byte-exact serializer, seed a fresh safekeeper's data directory from the backed-up WAL, and craft the control file so it reports the right position. Plus one more discovered subtlety: seed slightly past the pageserver's LSN so it re-derives prev_record_lsn from the streamed delta. Result: a restored plane where an INSERT survives a compute kill — full writable service from an off-cluster backup alone, automated.

The other experiment: an AI agent loop that reviews itself

This entire platform was built by AI agents — but the interesting part isn't the code generation, it's the process: a standing loop of plan → implement → test → blind review → plan again.

Every iteration, three fresh reviewer agents — a system designer, a DevOps/SRE, and an architect — score the platform 1–10 on maturity, ease of maintenance, and production reliability. Two rules made it work:

1. Reviewers are blind and disposable. They never see previous reviews, and they're retired after each round. When two independent reviewers hit the same finding, it's confirmed. When a reviewer implements their own finding, they're disqualified from scoring it.
2. The repo is the only memory. Findings live in issues, decisions in ADRs, numbers in a benchmarks file. A fresh agent reads its way to full context — nothing important lives in anyone's head (or context window).

The loop caught things I wouldn't have: an ADR whose header contradicted its own decision section (two blind reviewers found it independently), manifests that were merged but never applied to the live cluster ("grep-green, prod-red" — caught three times, which is why there's now a live-drift check in the test battery), and an alert drill that hung for one reviewer because Alertmanager silently deduplicates repeated test alerts for four hours.

The scorecard over five iterations tells the story better than any retro:

That ease-of-maintenance dip in iteration 4 is my favorite row: reliability was bought with new machinery (a failover watcher, a binary-format serializer), and the reviewers priced that debt instead of applauding. A scorecard that only goes up is a scorecard nobody should trust.

What I'd tell you to steal

• Separated-storage Postgres makes scale-to-zero almost free. The hard part was solved by Neon; the glue is a proxy and a scale call.
• Measure the pod path before touching the database. DNS caches and probe ticks will eat seconds while you micro-optimize milliseconds.
• Rehearse restores or you don't have backups. Ours surfaced a torn-index race, an unbounded RTO under WAL bloat, and the read-only gap — none of which a mc mirror cronjob would ever have told us.
• If AI agents build it, make them review it blind — and keep the memory in the repo. The loop's honesty mechanisms mattered more than the model's cleverness.

The repo is getknext-dev/scale-zero-pg — attribution and licenses in the README; everything above is drill-measured, with the receipts in docs/BENCHMARKS.md. Next up: binding it to knext so the app and its database sleep and wake together.