RTO

The Definition

 RTO (Recovery Time Objective):
 maximum acceptable time from DISASTER to SERVICE RESTORED.

 disaster at 14:00, RTO = 1 hour:
 users may endure (degraded or absent) service until 15:00.

 [service: ████████]✖disaster·······█████████
            │◄────── RTO budget ─────►│
            detection + decision + recovery + verification

 RTO answers HOW FAST YOU'RE BACK;
 RPO answered how much data survived. independent axes!

The Four Recovery Postures

              cost ▲
 COLD:   empty infra, restore from backups on demand
         RTO: hours–days. cheap. slow.
 WARM:   scaled-down stack running, replicas lagging;
         scale up + promote on disaster
         RTO: ~30min–hours
 HOT:    full-size standby mirroring production continuously
         RTO: minutes (cutover only)
 MULTI-ACTIVE: traffic already served from multiple regions;
         RTO: seconds (no failover — reroute)
         cost: highest

 pick per system tier:
 payments → hot/multi-active; internal tools → cold is FINE.

Where RTO Time Actually Goes

 the budget breakdown surprises people:

 detection & declaration:      5–30 min   (alerts, human call)
 DECISION to failover:         5–60 min   (politics! false alarms!)
 actual technical cutover:     5–15 min   (the easy part!)
 verification before traffic:  10–30 min  (don't cut to broken)

 the TECHNICAL work is rarely the bottleneck.
 unprepared DECISION-MAKING is.
 
 fixes:
 □ pre-authorized runbooks: criteria met → failover WITHOUT
   emergency committee formation
 □ automated failover for clear-cut scenarios (DB promotion)
 □ game-day rehearsals making decisions muscle memory

Testing RTO Honestly

 tabletop ≠ reality:

 □ GAME DAYS: actually failover in staging with stopwatch
 □ REGION EVACUATION DRILLS: real traffic shift, off-hours,
   full team — measure EVERY phase
 □ track achieved-RTO per drill; trend it like an SLI
 □ chaos hooks: partial failures (DNS-only? DB-only?) drilled
   separately from full-region events

 first real drill typically reveals 3–10× the expected time.
 that's the point of drilling instead of discovering it live.

Tier	Target RTO	Posture	Drill cadence
Revenue-critical	< 15 min	hot/active-active	monthly
Core product	< 1 h	warm	quarterly
Internal tools	< 8 h	cold	yearly

Interview Framing

“Region dies. When are you back?” scored shape: define RTO, posture table chosen per system tier, the time-breakdown insight (decision latency > technical cutover) with runbook/automation fixes, drill-based measurement honesty. Pairing RTO with RPO unprompted (“and we lose ≤N seconds of data”) shows both axes internalized — interviewers notice the pairing.