Disaster recovery you can rehearse (or: you don't have backups)

Part 4 of the scale-to-zero Postgres series.

Every review round of this platform flagged the same CRITICAL: no backups; this is the project-ending incident. Closing it taught me that the backup is maybe 20% of disaster recovery. The other 80% is what you learn the first time you actually restore — which is why the deliverable was never a CronJob. It was a rehearsal: a script that backs up the live system, rebuilds a complete storage plane from that backup in a throwaway namespace, and reads the data back. Run on demand, part of the test battery.

Here's everything the rehearsal caught that a backup job alone never would.

Finding 1: a backup isn't real until the storage says so

In a separated-storage Postgres, a committed write lives first in the safekeepers' WAL and only later in the pageserver's uploaded layers. A restore boots from the bucket — so a row that committed but hasn't been uploaded yet is simply not in your backup, no matter what the client saw.

The drill enforces this as its first honesty gate: write a marker row, then block until the pageserver's remote_consistent_lsn passes the marker's LSN, then back up. Operationally: never call data "backed up" on commit alone. Know your durability pipeline's stages and which one your backup captures.

Finding 2: mirrors capture races

At larger bucket sizes the restore intermittently failed on a torn index_part.json — the mirror had copied the pageserver's index mid-compaction rewrite. A backup that's a byte-for-byte mirror of a live system inherits the live system's in-flight moments. The drill grew a detect-and-retry; the durable fix (verify or order the index copy) went on the backlog as its own issue. Neither would exist without a rehearsal that ran often enough to hit the race.

Finding 3: the RTO you measured isn't the RTO you have

The first rehearsed restore took ~110 seconds. Months of drills later, a reviewer re-ran it and clocked over an hour — the live bucket had bloated to 13GiB of accumulated test WAL, and restore time scales with bucket size when you re-download across the internet. Nothing was broken; the RTO had silently decayed. The benchmark table now records the curve (`110s @2GB → 1076s @5GB →

60min @13GiB, unbounded-in-practice`), and WAL pruning got promoted from "hygiene" to "DR-readiness blocker." An RTO is a perishable measurement, not a property.

Finding 4: the read-only wall — and going under the API

The biggest catch: restores came up read-only. A read-write Postgres needs its safekeepers to confirm WAL continuity from the basebackup point, but freshly created safekeepers report flush_lsn 0/0 — and the Neon OSS release we pin ships no API to create a safekeeper timeline at an arbitrary LSN. The honest docs line for a while read: "promotion to writable is manual."

The breakthrough went under the API. The safekeeper's state lives in a small binary file — safekeeper.control, magic 0xcafeceef, version 9, crc32c trailer. Reverse-engineer it, write a byte-exact serializer, seed a fresh safekeeper's directory from the backed-up WAL, and craft the control file to report the correct position. Suddenly the safekeeper remembers a history it never lived, Postgres finds its continuity, and the restored plane boots read-write.

Two sub-discoveries en route, both classic distributed-systems texture:

• The pageserver had amnesia about prev_record_lsn on cold load, and Postgres refuses read-write without it. Fix: seed the safekeeper slightly past the pageserver's position so it streams a real WAL delta and re-derives the value.
• The pageserver only starts pulling WAL after a compute "kicks" the safekeeper. The promotion helper worked in early tests only because a crash-looping pod happened to be supplying kicks. The scripted flow now boots a primary deliberately, waits for catch-up, then does the final alignment.

End state, drill-proven: INSERT into the restored plane, kill the compute, and the row survives a fresh re-basebackup. Full writable service from an off-cluster backup alone — the promotion step itself costs ~134 seconds and is bucket-size independent.

Finding 5: generations, or how not to corrupt what you just saved

One more piece of tribal knowledge the drills forced onto paper: re-attaching a restored tenant must happen at a higher generation number than the original. The pageserver picks the newest index at-or-below its own generation — attach at generation+1 and it reads the old index and writes forward safely; attach at the same generation and you risk overwriting the very index you restored. This rule now lives in the runbook with the reasoning, because "mysterious integer must be incremented" is exactly the kind of knowledge that otherwise retires with whoever learned it.

The meta-lesson

Every finding above shares one shape: invisible until rehearsed. The consistency gate, the torn index, the RTO decay, the read-only wall, the walreceiver kick, the generation rule — none appear in a green CronJob log. If your DR plan hasn't been executed end-to-end recently, against real infrastructure, by someone following only the written runbook — then what you have is a backup-shaped object and a theory.

Next and last in the series: the 400ms warm tier — parking a database without breaking single-writer.