Backup Testing

The Graveyard Statistic

 industry surveys consistently find:
 large shares of backup RESTORES fail when actually attempted —
 missing data, wrong versions, broken dependencies,
 expired credentials, unknown procedures.

 backups fail SILENTLY: jobs report success, media fills,
 schemas drift, permissions rotate. nothing pages.
 the first full-stack test happens during the disaster.

 TESTING converts backups from hypothesis to evidence.

The Testing Ladder

 L1 AUTOMATED INTEGRITY: checksums, size sanity, job success
     → daily, cheap, catches silent rot
 L2 SINGLE-TABLE/FILE RESTORE: pull one object to staging,
     verify contents → weekly, automated
 L3 FULL-STACK RESTORE to isolated env: entire system from
     backups alone; app boots; smoke tests pass → monthly
 L4 TIME-TRAVEL TEST: PITR to specific timestamp;
     verify expected state exists → quarterly  
 L5 FULL DR GAME DAY: restore + failover + business flows,
     timed against RTO/RPO targets, humans on runbooks → 
     per quarter for critical tiers

 each level catches failures the previous can't:
 L1 misses logical corruption; L3 misses cross-service
 consistency; L4 proves PITR precision claims.

What Real Tests Check

 beyond "data came back":

 □ COMPLETENESS: all services' stores? config? secrets?
   (the forgotten bucket is always someone's favorite)
 □ CONSISTENCY: cross-service references valid
   (orders pointing at users that didn't restore = hell)
 □ APPLICATION COMPATIBILITY: schema versions match deployed code
 □ PROCEDURE ACCURACY: runbook steps ACTUALLY work as written;
   drift between docs and reality is the norm — fix docs
 □ TIMING: measure phases against RTO budget; trend over time
 □ ACCESS: credentials in the runbook still valid? (rotation!)

Making It Sustainable

 automation carries the load:

 □ scheduled restore-jobs into disposable environments
   (IaC builds the target; tests run; environment dies)
 □ results DASHBOARD: last-verified date PER BACKUP SET —
   anything untested >90 days shows red
 □ game days human-led but scripted; findings feed backlog
 □ test PRODUCTION-LIKE scale occasionally: restoring 2TB
   behaves differently than 2GB
 □ include the HUMANS: who executes at 3am? they run drills.

 culture metric: nobody should be able to say
 "we've never actually tried restoring that."
 if true today — schedule it this sprint.

Interview Framing

“How do you know your backups work?” scored shape: reject assumption-based answers immediately, present the ladder (integrity → single-object → full-stack PITR → game day), forgotten-components checklist (config/secrets/cross-service consistency), automation-with-dashboard sustainability, RTO measurement during drills. This question filters relentlessly: candidates who’ve lived an untestable-backup incident answer with scar-tissue specificity.