Failure Domains

What a Failure Domain Is

 the set of components that fail TOGETHER when something breaks:

 - one process:        crash kills its requests only
 - one host:           hardware/kernel failure takes its containers
 - one AZ:             power/network event blacks out a zone
 - one region:         regional disaster (rare, real)
 - one shared cluster: noisy-neighbor resource exhaustion
 - one shared DB:      schema lock / overload = everyone waits

 design question: WHEN X FAILS, WHAT ELSE DIES WITH IT?
 answer honestly per layer; then SHRINK the answers.

Mapping and Shrinking Domains

 inventory your hidden shared fates:

 SHARED COMPONENT         DOMAIN IT CREATES       SHRINK BY
 single cache cluster     entire platform         tiered/partitioned caches
 one message broker       all async flows         broker clusters/segments
 single auth service      every request           replicated + fallback tokens
 one DB instance          everything using it     sharding, replicas
 same-AZ deployments      zonal outage            spread across AZs ALWAYS
 one k8s namespace        its workloads           namespace/node-pool splits

 THE CARDINAL RULE: multi-AZ by default for anything critical.
 zonal failure should be a NON-EVENT: traffic shifts,
 latency wiggles, nobody pages. if it isn't, that's the roadmap.

Blast Radius Thinking in Design Reviews

 for each component ask three questions:

 1. what does ITS failure take down?    (domain size)
 2. how likely is that failure?          (frequency)
 3. how fast do we detect+recover?       (dwell time)

 RISK = size × frequency × dwell

 attack the biggest products first:
   huge domain + slow recovery = priority refactor
   small domain + auto-recovery = acceptable residual

 document the map — tribal knowledge of "what shares fate
 with what" evaporates exactly when you need it.

Domain Boundaries as Architecture

 boundary choices ARE resilience architecture:

 - CELL-BASED: full stacks per cell; one cell's failure
   affects only its users (own lesson)
 - SHARDING as failure isolation: user-shards mean one DB
   shard's crash = fraction of users degraded, rest fine
 - PER-TENANT isolation: noisy tenant can't starve others
 - GEOGRAPHIC independence: regions serve independently
   even disconnected (DR lessons build on this)

 tradeoff honesty: smaller domains cost more infrastructure,
 more operational surface, more cross-boundary complexity.
 right-size: shrink domains where blast radius × likelihood hurts.

Interview Framing

“What happens when this availability zone fails?” scored shape: enumerate the domain contents honestly (instances, caches, DB primary!), show traffic-shift behavior and capacity math (N+1 zones), name hidden shared-fates found in the design (single broker/cache) with fixes, connect to cell/shard isolation as structural strategy. The follow-up hiding inside every design interview is domain-mapping fluency — volunteer it.