Regional Failover

The Timeline of a Region Loss

 T+0:     region degrades/dies
 T+0–5m:  DETECTION — synthetic probes, error-rate alarms,
          cloud-provider status (unreliable early!)
 T+5m:    DECISION — is it regional? partial? failover-worthy?
          (hardest step; false positives are costly too)
 T+10m:   EXECUTE — traffic reroute + data promotion
 T+15m:   VERIFY — business flows green, capacity checked
 T+15m+:  OPERATE degraded; monitor the new single-region risk
 LATER:   FAILBACK once home region returns

 every phase needs pre-built machinery. none can be
 improvised at 3am to target-class numbers.

Detection: Trusting the Right Signals

 signal quality ranking:

 ✓ EXTERNAL SYNTHETICS probing from OTHER regions
   (your monitoring INSIDE the dead region dies with it!)
 ✓ cross-region error/latency correlation dashboards
 ✗ provider status pages: lag real-world truth by many minutes
 ⚠ customer reports (fast but unstructured)

 automated-vs-human trigger:
   clear-cut criteria (error rate >X% from all probes
   for >N minutes) → auto-failover for defined scenarios;
   ambiguous → human with RUNBOOK, not committee.

The Execution Checklist

 □ TRAFFIC: DNS weights / global LB pool switch
   (TTLs pre-lowered on failover-able records!)
 □ DATA: promote replicas (AP) or confirm replication health (AA)
 □ QUEUES: consumers activate in survivor region
 □ SECRETS/CONFIG: present in target region (pre-staged!)
 □ CAPACITY: survivor region sized for FULL load already?
   (warm capacity math; autoscaling headroom verified)
 □ DEPENDENCIES: third-party services region-pinned?
   (some SaaS endpoints need config flips too)
 □ CACHES: expect cold-start stampede → pre-warming plans,
   shedding posture ready

Failback: The Sequel Nobody Plans

 returning home safely:

 1. home region restored & VERIFIED stable
 2. REVERSE REPLICATION: failover-region's writes → home
    (this direction often never built until needed!)
 3. drain traffic gradually during low-peak window
 4. verify consistency before final cutover
 5. keep runbook updated with what reality taught

 teams burned by failback skip step 2 and either
 lose writes or fear them into staying failed-over forever.
 build reverse-replication INTO the original design.

Phase	Failure mode if unplanned
Detection	blind (monitoring died with region)
Decision	hours lost in committee
Execution	missing secrets/config in target
Verification	cut over to half-working stack
Capacity	survivor region melts under doubled load
Failback	write loss or permanent drift

Interview Framing

“US-East-1 is down. Walk me through your next hour” scored shape: phase-by-phase timeline with pre-built mechanisms named per phase, external-synthetic detection insight, warm-capacity math for survivors, split-brain guard reference, failback-with-reverse-replication closing. Narrating the DECISION phase honestly (“ambiguous signals get a human with a runbook”) reads as operational experience, not theory.