Restore

The Uncomfortable Truth

 backups without RESTORE practice are insurance policies
 written in invisible ink:

 - restore takes 9 hours, not 30 minutes (discovered live)
 - restored schema missing latest migrations (app won't start)
 - credentials/connections point to old infra (manual surgery)
 - data consistency across services unverified

 THE METRIC THAT MATTERS: time-to-working-system.
 not "backup exists." time until USERS are served correctly.

The Restore Playbook

 documented, REHEARSED, versioned:

 1. DECLARE:    who calls it, severity level, comms channels
 2. ISOLATE:    restore to staging/replacement infra,
                never overwrite possibly-good data first!
 3. RESTORE:    base snapshot → apply incrementals → WAL to point
 4. VERIFY:     row counts, checksums, app-level probes:
                can a test user log in? check out? data sane?
 5. CUTOVER:    DNS/pool switch; keep old env for forensics
 6. RECONCILE:  transactions during outage window — 
                what happened after backup point? replay? accept loss?
 7. REVIEW:     actual-vs-target RTO; update playbook with reality

 step 3's mechanics rehearsed quarterly = boring.
 unrehearsed = improvisation theater under pressure.

The Speed Math

 restore throughput determines feasibility:

 2TB database, restore at 200MB/s:      ~3 hours just copying
 + index rebuilds (often LONGER than copy!)
 + application warmup/cache refill
 
 options when too slow for your RTO:
 □ WARM STANDBY already replicating (restore becomes cutover)
 □ SNAPSHOT-attach instead of copy-then-rebuild
 □ PARTIAL restore: critical tables first, rest lazily
 □ FREQUENT drills keep paths optimized (tooling improves)

 write down YOUR numbers: size, throughput, rebuild time,
 total. if you can't, you don't know your RTO.

Verification Beyond “It Started”

 depth of restore checks:

 L0: process boots                     ← necessary, insufficient
 L1: schema matches expected version
 L2: row counts within tolerance vs pre-incident metrics
 L3: business-flow smoke tests pass (login→browse→checkout)
 L4: cross-service consistency verified (orders↔payments agree)
 L5: spot-audit against external truth (email receipts, analytics)

 automate L0–L3 into the drill pipeline;
 humans judge L4–L5. restore isn't done when data loads;
 it's done when BUSINESS flows work on it.

Interview Framing

“Walk me through recovering from yesterday’s bad migration” scored shape: PITR-to-pre-migration-point, isolate-then-cutover ordering (never blind-overwrite), verification ladder beyond boot-success, reconciliation of post-backup transactions, and the practiced-not-improvised emphasis with drill cadence. Restores reveal whether backups were engineering or decoration — answer like someone who’s drilled it.