Origin Failover

The Design Goal

 origin outages are WHEN-not-IF. CDN posture:

 users should experience:
   best case:  nothing (stale content serves)
   middle:     partial freshness degradation  
   worst case: explicit maintenance page — never raw errors

 "the site is slow" is recoverable.
 "the site shows stack traces" is a brand event.

Layer 1: Serve Stale

 the highest-value headers in resilience engineering:

 Cache-Control: max-age=60, stale-while-revalidate=300,
                             stale-if-error=86400

 normal:            fresh for 60s
 background refresh: next 5 min served instantly while updating
 ORIGIN ERROR:      up to 24 HOURS of cached content keeps serving

 during an origin outage:
   hit-rate-95% site → 95% of pages fully working from cache
   users browse; only uncached paths notice. incidents become
   non-events through one config line set months earlier.

Layer 2: Multi-Origin Failover

 configure fallback chains at the CDN:

 primary:   us-east.origin.app      (healthy? serve)
 fallback1: us-west.origin.app      (primary down → try)
 fallback2: static-s3-bucket        (last resort: static snapshot)

 health-based selection with hysteresis (no flapping):
 - probe /health every few seconds per origin group
 - N consecutive failures → demote; M successes → restore
 
 cross-region origins need REPLICATED data behind them —
 failover to an empty region helps nobody (DR lessons apply).

Layer 3: Negative Caching Discipline

 protect a struggling origin from retry storms THROUGH the edge:

 error responses get short TTLs too:
   502/503/504 → cache 5-30s at edge
   404s        → cache minutes (also penetration defense)

 without negative caching: every user retry = fresh origin hit =
 dying origin gets MORE traffic. edges must ABSORB failure,
 not amplify it.

Layer 4: Graceful Static Fallback

 prepare a "break-glass" static site:

 - core pages + status messaging pre-built to object storage
 - wired as final failover origin
 - feature-detect: app JS detects API failures → degraded mode UI
   (read-only browsing, queued actions, clear messaging)

 companies with this ready convert outages into
 "minor slowdowns" publicly. it's cheap insurance.

Monitoring the Right Things

Signal	Meaning
Edge hit rate drop	Origin trouble OR cache purge accident
Origin fetch error rate	Direct origin health view
Stale-serving volume	How much resilience is ACTIVE right now
Time-to-failover	Multi-origin switching speed

Alert on stale-serving spikes: that metric firing means your safety nets are catching something — investigate even if users haven’t noticed yet.

Interview Framing

“Origin region goes down mid-launch” scored arc: stale-if-error serving as first line (quantify via hit rate), multi-origin chain with replicated data caveat, negative caching against amplification, static break-glass as last resort. Framing the outcome as “freshness degradation instead of outage” shows you understand what edge resilience actually buys.