Load Shedding

The Overload Death Spiral

 traffic exceeds capacity. naive service behavior:

 accept everything → queue grows → latency climbs →
 clients time out → clients RETRY → MORE load →

 [load]  ▁▂▃▅▇██ overload
 [served] ▁▁▁▁▁ collapsing toward zero

 accepting work you cannot finish is WORSE than refusing:
 everyone gets timeouts instead of some getting fast answers.

 LOAD SHEDDING: deliberately refuse excess EARLY,
 protect capacity for what you CAN serve well.

What to Keep, What to Shed

 priority classes, decided in advance:

 P0 keep-at-all-costs:   checkout, payment, auth
 P1 degrade-gracefully:  search, product pages
 P2 shed-first:          recommendations, analytics beacons,
                         prefetch, non-critical background

 shedding rules by request properties:
 - authenticated paying user > anonymous crawler
 - read-your-cart > speculative recommendations
 - small/fast requests > giant slow ones (fairness)
 - NEW sessions > deep pagination scrapes

 anti-goal: FIFO fairness during overload.
 fairness means EVERYONE suffers; triage means the
 important things survive.

The Mechanics

 where and how to refuse:

 EDGE (best): LB/gateway rejects before consuming app
   capacity:    concurrency limits, rate rules per class
   response:    429/503 + RETRY-AFTER (politeness matters —
                uncoordinated client retries = next incident)

 APP-LEVEL (adaptive):
   measure saturation: queue depth, CPU, latency drift
   above soft threshold → start shedding P2
   climbing → shed P1 extras, serve cached/stale
   hard limit → only P0 passes

 ADAPTIVE signals worth using:
   concurrency in-flight vs budget (simple, effective)
   latency SLO burn (serve until SLO at risk, then shed)
   little's-law backpressure: L = λ×W exceeding bounds

Designing the Client Experience

 a shed request must fail WELL:

 □ explicit status: 429/503 with Retry-After header
 □ idempotent-safe retry guidance (clients WILL retry)
 □ degraded modes pre-built: cached pages, static fallbacks,
   "we're busy" states that look intentional
 □ mobile clients: exponential backoff BUILT INTO SDKs,
   jittered — or your shed becomes their hammer
 
 measure shed metrics: rate per class, post-shed recovery time,
 revenue-path survival ratio. THE metric of overload design:
 did P0 stay healthy while P2 drowned? ✓

Interview Framing

“Flash sale drives 20× normal traffic; site melts” scored shape: admit capacity ceiling honestly, priority classification table (P0 commerce paths protected), edge-level shedding mechanics with Retry-After, adaptive thresholds named, degraded-mode UX designed, success metric stated (“checkout stays <500ms while browsing sheds”). Refusing gracefully IS the architecture answer here — hero-scaling promises are not.