Automatic Rollback

Why Automate the Revert

 human-in-the-loop rollback adds MINUTES when seconds exist:

 [gate breach 14:00:00]
   human notices pager     +2–10 min   ← damage accruing
   human decides           +2–15 min   ← meetings at 3am
   human executes          +1–5 min

 automated path:
   gate breach → threshold confirmed → revert executes → alert
   total: seconds-to-a-minute. damage window collapses.

 the philosophy: machines handle the UNAMBIGUOUS case fast;
 humans investigate afterward with the system already safe.

Trigger Design

 what fires automatic rollback:

 □ HARD signals (auto-revert immediately):
   error-rate delta vs control > threshold for N minutes
   p99 latency delta > bound
   crash-looping new pods / readiness never passing
   health-check failure across new instances

 □ SOFT signals (alert-only, human decides):
   business-metric wobble (could be external factors)
   slow-burn trends (memory creep)
   single-instance anomalies (maybe just that host)

 hard/soft split prevents both extremes:
 auto-rollback on every blip = deploy thrash;
 requiring humans for everything = slow disasters.

 confirmation windows matter: require breach SUSTAINED
 (e.g. 3 of 5 minutes) so single-sample noise doesn't thrash.

Guardrails Against Automation Gone Wrong

 auto-rollback can misfire; fence it:

 □ RATE LIMIT rollbacks per service per hour (a genuinely
   broken pipeline shouldn't flap 20 reverts)
 □ CIRCUIT to manual mode: repeated auto-reverts on the same
   build → freeze deploys + page humans (something systemic)
 □ DISTINGUISH deploy-caused from ambient incidents:
   correlate breach timing against THIS rollout specifically;
   a region outage shouldn't "roll back" an innocent release
   (compare treatment-vs-control cohorts — canary framing
   naturally handles this ✓)
 □ AUDIT TRAIL every trigger: what fired, which metric,
   what values — postmortems depend on it
 □ REHEARSE the automation itself in game days:
   inject a bad build deliberately; verify it self-arrests.

Failure mode	Guardrail
flapping reverts	rate limits + confirmation windows
innocent releases punished	cohort comparison
silent wrongness	audit + alert-on-action
stale rollback target	last-known-good pinning discipline

Interview Framing

“Design deployment safety that doesn’t depend on who’s on call” scored shape: latency arithmetic motivating automation, hard-vs-soft trigger table with confirmation windows, guardrail set (rate limits, cohort correlation, audits), rehearsal requirement stated. This question tests whether you trust automation WITH judgment boundaries — the guardrails are the answer, not the automation itself.