Design for Failure

The Mindset Shift

Single-server thinking treats failure as an anomaly to prevent. Distributed-systems thinking treats it as weather — continuous, partially predictable, survivable by design. At any moment, in any system of meaningful size, something is failing:

 AT 10,000 MACHINES (Amazon's Bezos-era math, still directionally true)

 1,000 machines/yr × hard-drive failures      ~ 1 failing disk every few hours
 RAM bit-flips, NIC flaps, power blips        constant background hum
 network partitions between racks             weekly-ish somewhere
 a deploy gone wrong                          self-inflicted, every week

 CONCLUSION: "will it fail?" is the wrong question.
 "what happens WHEN each part fails?" is the design question.

From Prevention to Containment

You cannot prevent failure at scale; you can only decide its blast radius and user experience:

 UNDESIGNED FAILURE                 DESIGNED FAILURE

 DB slows → threads pile up →       DB slows → circuit breaker trips →
 app threads exhaust → whole app    cached/fallback reads served →
 returns errors EVERYWHERE          degraded but alive; only affected
 users retry → retry storm →        calls degrade; bulkheads contain
 total outage                       the rest

The right-hand path uses named mechanisms — each covered in depth later in this course:

Mechanism	Failure it contains
Redundancy + failover	Single node/AZ death
Circuit breakers	Slow dependencies cascading
Bulkheads	One tenant/pool exhausting shared resources
Timeouts + retries with backoff	Hanging calls; transient faults
Graceful degradation	Dependency loss becoming user-visible outage
Load shedding	Overload melting everything equally

Design Questions That Force Failure Thinking

Walk any architecture diagram box-by-box asking:

1. What if this dies? — Is there a second one? How does traffic know?
2. What if this is slow? — Do callers time out cleanly or hang forever?
3. What if this returns wrong data? — What validates? What’s the blast radius?
4. What if this comes back after being dead? — Thundering herd? Stale state?

A box with no answer to #1 is a single point of failure — found in five minutes of questioning instead of five months of incidents.

Failure Domains and Independence

Redundancy only helps when replicas fail independently:

 USELESS REDUNDANCY                  REAL REDUNDANCY
 App A + App B on same rack,         App A in AZ-1, App B in AZ-2,
 same power, same ToR switch         different power feeds;
                                     DB primary AZ-1, replica AZ-2,
 both die together = one failure     async replication, promoted on
 wearing two costumes                loss with known RPO trade-off

Independence axes: hardware, power, network path, software version, even operator (change freezes during shared deploys).

The Cultural Piece

Systems that survive failure are practiced at failing: chaos engineering injects faults deliberately; game days rehearse responses; postmortems are blameless so truth surfaces; error budgets make reliability a budgeted feature rather than an aspiration. Netflix’s Chaos Monkey — killing production instances on purpose — is the canonical expression: prove resilience continuously rather than assume it.

Interview Framing

“Design for failure” is graded through reflexes: after drawing any component, strong candidates immediately annotate its death story (“if this Redis dies, requests fall through to DB — sized to absorb the miss storm”). A diagram where every box has a stated failure behavior reads as experience.