Cache Breakdown

The Scenario

 stampede/breakdown distinction:

 stampede:   ONE hot key expires → herd on one query
 BREAKDOWN:  the ENTIRE cache layer fails (redis restart, network
             partition, OOM) → 100% of reads convert to db load

 if your db serves 10x its designed peak while already missing
 its warm caches... that's an outage chain, not a slowdown.

The Cascade

 [cache dies] ──► all reads hit db
              ──► db latency rises 5-20x
              ──► app timeouts on db
              ──► retries multiply load
              ──► connection pools exhaust
              ──► TOTAL outage (both layers now)

 breakdowns are dangerous precisely because the cache was
 sized assuming it would WORK — remove it and every
 downstream capacity assumption inverts.

Defense: Fail Behavior Design

 decide EXPLICITLY what happens when cache is unreachable:

 FAIL OPEN (most reads):
   catch cache error → proceed to db with:
   - reduced request budget (shorter timeout)
   - concurrency limiter on origin queries
     (semaphore: max 200 concurrent fills site-wide)
   
 FAIL CLOSED (only for correctness-critical):
   auth token revocation checks etc. where stale = security hole
   → error, but scoped to that check only

 never let ONE cache's failure take unrelated endpoints down:
 bulkhead the cache dependency per feature.

Defense: Load Shedding at Origin

 when db can't serve full miss traffic, choose WHO waits:

 - prioritize by endpoint class: checkout > profile > recommendations
 - serve degraded responses (cached-stale-if-error, default pages)
 - admission control: reject excess EARLY with 503+Retry-After
 
 a 60%-success site beats a 0%-success site. shedding is
 how you keep the 60%.

Defense: Warm Recovery

 cache restarts empty → first minutes are the danger window:

 - PROGRESSIVE WARMING: preload top-N keys before serving traffic
   (know your hot set from analytics; keep a snapshot)
 - RAMP TRAFFIC: health-gate pods until warmup completes
 - WILDCARD TTL STAGGER: after cold start, jitter refills so
   expiry doesn't re-synchronize later

 recovery plan is part of cache design; cold starts are
 scheduled events (deploys), not surprises.

Resilience Posture Summary

Layer	Breakdown defense
Cache client	Timeouts + fail-open + circuit breaker
App tier	Per-dependency bulkheads, degraded modes
Origin db	Concurrency limits, load shedding
Ops	Warm snapshots, staged restarts

Interview Framing

“Redis cluster goes down mid-morning — walk me through your system’s behavior.” Scored arc: name total-miss conversion as the mechanism, fail-open with bounded concurrency (not naked db flood), shed load by priority, warm-recovery plan post-restart. The differentiator is treating cache as a DEPENDENCY WITH FAILURE MODES, not invisible infrastructure.