2026.07.04 · 9 min read · postgres

The wake-on-connect gateway: speaking just enough Postgres

A proxy that wakes databases must speak the Postgres wire protocol — but only the first 100 bytes of it. Everything after that is knowing when NOT to touch the stream.

Ahmed El Banna
Ahmed El Banna
Technical Leader · Full-Stack Engineer

Part 2 of the scale-to-zero Postgres series. Part 1 covers the architecture; this one goes inside the gateway.

The whole scale-to-zero product is experienced through one component: a Go proxy that sits where a Postgres server should be. When a client connects to a sleeping database, the gateway wakes it, waits, and splices the two sockets together. Done well, the app can't tell the database ever slept.

The design principle that kept it small: speak just enough wire protocol.

The first 100 bytes

A Postgres client opens a TCP connection and sends one of four initial packets, each framed as a big-endian length plus a magic number:

That's the entire protocol surface the gateway parses. The database parameter doubles as the routing key. After the startup message is read, the gateway resolves the target compute, ensures it's awake, replays the raw startup bytes to the backend, and becomes a dumb io.Copy in both directions — authentication, queries, COPY, everything flows through untouched.

TLS lives at the same seam: if a cert is mounted, the gateway answers SSLRequest with 'S' and wraps the client socket in tls.Server before reading the startup message. Half-configured TLS (cert without key) refuses to boot rather than silently serving plaintext — config that can fail should fail loudly at startup, not at 3am.

Waking is easy. Knowing when it's awake is not.

The wake itself is one Kubernetes API call — scale a Deployment from 0 to 1. The gateway then polls a TCP connect until the backend accepts. Simple. Wrong.

A freshly started Postgres accepts TCP before it can serve. During startup it answers the handshake with FATAL: the database system is starting up (SQLSTATE 57P03) — and a naive proxy pipes that straight to the client. In our drills this hit roughly 40% of cold wakes. The fix: after replaying the startup packet, the gateway peeks at the backend's first reply. If it's a 57P03 ErrorResponse, close, wait, reconnect, replay, peek again — until the deadline. The client, still held on the front socket, never sees any of it.

The same peek loop handles a nastier sibling: a backend that accepts and then drops the connection with no reply at all (a terminating pod during a scale-down race). EOF-with-zero-bytes is treated like 57P03 — retry the wake — while a read timeout after bytes flowed means "slow but alive, start piping."

Sleep is a distributed-systems problem wearing a timer costume

Scaling to zero sounds like the easy half: no connections for N seconds, scale to 0. Three production bugs later, I disagree.

Bug 1 — the fleet lies. With two gateway replicas, each counts only its own connections. Gateway A, seeing zero, would happily kill a database that gateway B's client was actively querying. The fix is peer-aware idle: before sleeping, a gateway sums active_connections across its siblings (label-selected pods, scraped directly) and refuses to sleep unless the fleet is at zero. Any scrape error also refuses — when unsure whether someone is using the database, the answer is never "kill it."

Bug 2 — the TOCTOU window. Even fleet-wide-zero can change between the check and the scale-to-zero API call. A connection that arrives mid-call finds a dying database. The heal: re-check counts immediately before the call, and after a successful sleep, check once more — if someone arrived during the call, wake it right back. The arriving client is already inside the gateway's own wake-retry loop, so recovery is seamless.

Bug 3 — releasing the connection slot too early. The gateway caps concurrent connections (excess gets a clean SQLSTATE 53300 instead of an OOM). My first cap released the slot when the handler function returned — but the handler returns as soon as the pipe goroutines start. Slots freed while connections lived. The fix is a sync.Once release wired to the connection's Close, because every code path — handshake error, pipe teardown, timeout — closes the socket exactly once.

Configuration that can't drift

One postmortem lesson worth stealing outright. The gateway originally read its env config through a whitelist of known keys. Someone (me) later added GW_IDLE_MS to the deployment but not the whitelist — the value silently fell back to a compiled default 5× larger, and we spent an hour chasing a "broken" idle timer that was working exactly as configured. The whitelist is gone: every GW_* var passes through, and the startup log prints the effective config.

The companion habit: every operational claim became an executable drill. _verify-wake.sh proves the full 0→1→0→1 loop; _verify-ha.sh holds a connection across the idle window through one gateway while killing the other; the drills run against the live cluster, not a mock. When a reviewer later ran one and got a different number than the docs claimed, that mismatch was itself the finding.

What I'd keep, what I'd warn about

Keep: the minimal-protocol stance. Ten months of Postgres wire evolution can't break a proxy that only reads the startup packet. Keep: fail-safe bias everywhere the gateway is unsure (peer errors, half-config, dying backends). Warn: an always-on proxy is now on your data path — cap its connections, watch its memory, and treat its idle logic as the distributed-systems code it secretly is.

Next in the series: choosing the storage foundation with a bake-off, not a debate.