Cascading Failures

Anatomy of a Cascade

 t0:   payments service slows (GC pause, bad deploy...)
 t1:   orders' calls to payments hang → orders threads pile up
       (missing timeouts!) → orders saturates
 t2:   frontend's calls to orders hang → frontend pools exhaust
 t3:   health checks time out → LB pulls healthy-ish instances
       → remaining instances get MORE load → deeper failure
 t4:   retry storms from every client multiply traffic
 t5:   ENTIRE PLATFORM down. root cause? one slow service.

 [payments]──slow──►[orders]──hangs──►[frontend]──►[users]
        thread exhaustion propagates left to right;
        retries propagate right to left. pincer.

The Amplifiers

 every cascade is powered by the same suspects:

 1. MISSING TIMEOUTS: hangs instead of fast failures
    → resource exhaustion spreads
 2. UNBOUNDED QUEUES/POOLS: absorb pressure silently,
    fail catastrophically later
 3. RETRY STORMS: multiplied load on struggling deps
    (no jitter, no caps, layered retries)
 4. HEALTH-CHECK FLAPPING: LB churn during brownouts
    redistributes load onto victims
 5. SYNCHRONOUS CALL CHAINS: deep fan-out = wide blast radius
 6. SHARED RESOURCES: one DB/cache pool for everything
    = single point of contagion

The Defense Stack

Layer	Defense	Kills which amplifier
Calls	timeouts everywhere + deadline propagation	hangs
Clients	capped+jittered retries, ONE layer only	storms
Dependencies	circuit breakers + fallbacks	calling into fires
Resources	bulkheads per class	shared exhaustion
Overload	shedding + backpressure	death spiral
Topology	async messaging between domains	chain depth
Deployment	canary + auto-rollback	bad-code ignition

 no single defense suffices; cascades exploit gaps.
 layered resilience = each amplifier has a dedicated killer.

Detecting a Cascade in Progress

 signature metrics:

 - LATENCY CORRELATION across unrelated services
   (everything degrading together = systemic, not local)
 - THREAD/POOL saturation climbing in sequence downstream→upstream
 - RETRY-RATE spikes multiplying request volume
 - HEALTH-CHECK flapping counts

 incident response for cascades:
   shed load FIRST (break the spiral),
   then trip breakers manually if needed,
   THEN hunt root cause. order matters —
   debugging mid-spiral wastes minutes you don't have.

Interview Framing

“One service’s slowdown brought down the platform” scored shape: trace propagation through missing timeouts/thread exhaustion/retry storms explicitly, present the defense-stack table mapped to amplifiers, state the incident-order principle (shed → break → diagnose), mention topology-level fix (async boundaries) as structural prevention. This question tests systems thinking about FAILURE INTERACTION — the answer is always plural defenses.