Fault Injection

Injection Primitives

 every real-world failure reduces to a few primitives:

 LATENCY:    delay calls (+200ms, +5s)      → timeout/breaker tuning
 ERRORS:     return 500s at rate N%         → retry/fallback paths
 CRASH:      kill processes/containers      → self-healing, drains
 PARTITION:  drop network between nodes     → quorum, split-brain
 SATURATION: CPU/mem/disk/conn-pool stress  → degradation under load
 CORRUPTION: malformed responses/data       → validation layers
 TIME:       clock skew/jumps               → ordering, TTL logic

 combinations matter: latency-then-errors mimics real
 dependency death better than either alone.

Mapping Injections to Claims

 architecture claims → verifying injections:

 CLAIM                              INJECTION THAT TESTS IT
 timeouts are set correctly         latency injection until breakers trip
 circuit breakers open+recover      sustained error injection, then heal
 fallback content serves            hard dependency blackout
 retries don't amplify              error injection under load; measure
                                    total request multiplication
 queue consumers drain after crash  consumer SIGKILL mid-batch;
                                    verify no loss/dup corruption
 DB failover is transparent         kill primary; measure write stall
 cache stampede protection          flush cache at peak traffic shape
 rate limits shed gracefully        saturation ramp to 10× traffic

 if a claim has NO injection that verifies it,
 it isn't an engineering claim. it's a wish.

Where to Inject

Layer	Tools/mechanism	Catches
Unit tests	mock failures, fault-injecting fakes	logic branches cheaply
Integration	toxiproxy-class proxies	client resilience code
Platform (k8s)	chaos-mesh/litmus CRDs	pod/network/volume faults
Infrastructure	AWS FIS, firewall rules, iptables	AZ/instance-level claims
Application	env-flag-driven fault hooks	precise business-path faults

 start low (cheap, safe), climb only as confidence grows.
 application-level hooks give surgical precision for
 business-critical paths that generic tools can't reach.

Reading Results Honestly

 experiment outcomes:

 ✓ system behaved per hypothesis → document, increase radius
 ✗ weakness found → incident-grade finding:
     file with severity, fix, RE-RUN to verify fix
 ⚠ flaky/inconclusive → tighten measurement before re-running

 the metric that matters across ALL experiments:
 user-visible steady-state during injection.
 internal heroics that still degrade users = failed test.

 institutionalize: findings feed the backlog with same
 priority language as production incidents — because
 they ARE incidents that happened in daylight instead.

Interview Framing

“Prove this checkout flow survives payment-gateway brownouts” scored answer: specific injection plan (latency ramp then errors), claims-to-injection mapping shown, measurement of USER-facing metrics not internals, abort conditions, re-run-after-fix loop. Fault-injection fluency closes the resilience-story arc: designs earn trust through deliberate verification, and this is the vocabulary of that verification.