Chaos Testing in the Pipeline

From Game Days to Pipeline Gates

 chaos engineering's lesson covered experiments; this is its
 CONTINUOUS form — resilience verified per release:

 traditional: annual game day, manual, heroic
 pipeline-grade: automated fault scenarios in staging (and
   carefully scoped prod) run WITH every significant deploy

 [deploy candidate]──►[functional tests]──►[chaos suite]
                                              │
        inject: dep latency, instance kill, net partition
        assert: fallbacks engaged, SLOs held, recovery clean
                                              │
                                     gate: pass → promote

 resilience becomes a TESTED PROPERTY of each build,
 not an architectural claim from a doc nobody re-reads.

The Standard Scenario Suite

 baseline pack most services should pass per release:

 □ DEPENDENCY BLACKOUT: primary downstream returns 100% errors
   → breaker opens, fallback serves, error-budget burn bounded
 □ LATENCY INJECTION (+2s): timeouts fire correctly; no thread
   pile-up; retry budgets respected
 □ INSTANCE KILL mid-deploy: drain clean; zero client errors
   (zero-downtime drill automated!)
 □ CACHE FLUSH at load: stampede protection holds; DB survives
 □ NETWORK PARTITION to replica set: quorum behavior correct,
   no split-brain writes
 □ CLOCK SKEW +5min: ordering/TTL logic doesn't corrupt

 each scenario = hypothesis + injection + assertion + abort.
 encode ONCE; every future release inherits the check.

Assertions That Matter

 chaos suites fail on the WRONG things unless assertions
 are user-centric:

 ✓ assert CLIENT-visible health:
    success rate ≥ threshold DURING injection window
    p99 within budget (degraded-but-bounded)
 ✓ assert RECOVERY time after fault removal (back to
    baseline within N minutes — no limping)
 ✓ assert SAFETY: no duplicate side effects, no lost writes
 ✗ asserting internals only ("breaker state flipped") —
   breakers can flip while users still suffer; measure USERS

 flakiness discipline applies double here: a chaos test that
 fails randomly teaches teams to ignore chaos results.
 quarantine + fix like any flaky suite, or credibility dies.

Scoping Safety

Environment	What runs	Guardrails
PR-time staging	core pack, tiny blast	free-for-all
pre-prod full clone	full suite + soak-chaos	scheduled windows
production	narrow injections on canary cohorts	business-calendar aware, auto-abort

 production chaos stays SCENARIOS-NOT-SURPRISES:
 only previously-passed-in-staging injections, minimum viable
 blast radius, automatic abort on user-metric breach.
 the pipeline's job is making prod-chaos rare and boring —
 because staging caught it first.

Interview Framing

“How do you KNOW your resilience patterns still work after six months of changes?” scored shape: continuous-chaos-as-gate model drawn, standard-scenario suite listed with user-centric assertions emphasized, three-environment scoping table, flakiness-credibility point included. This closing question grades whether resilience was ENGINEERED as a maintained property or DECLARED once and forgotten — pipelines that verify beats documents that promise.