Rollback

Rollback Speed Is an Architecture Property

 mean-time-to-recovery is dominated by detection + DECISION +
 rollback mechanics. optimize all three:

 [bad deploy 14:00]──detect──►──decide──►──revert──►[good]
                     minutes      minutes     seconds?

 target: bad build's damage window ≤ minutes.
 that requires: gates catching it, authority pre-delegated,
 and a ONE-COMMAND revert. anything slower is architecture debt.

 kubernetes-native: rollout undo (previous manifest reapplied)
 blue-green: flip traffic back
 canary/mesh: weights to zero
 classic VM fleets: redeploy previous artifact — SLOW;
   keep last-known-good artifacts hot for this reason

The Stateful Complications

 code rolls back; DATA does not automatically:

 □ SCHEMA MIGRATIONS: if v2 migrated schema, old v1 code may
   break on new schema → expand-contract discipline makes
   rollbacks SAFE by keeping migrations backward-compatible
   across one version window ✓ (own lesson)
 □ NEW DATA written by v2: formats/fields v1 can't read →
   tolerant readers + versioned payloads absorb it
 □ CACHE pollution: v2 wrote new-format cache entries →
   flush affected keys or namespace caches per version
 □ EXTERNAL SIDE EFFECTS: emails sent, payments captured —
   rollback doesn't unsend. incident response ≠ just revert;
   business reconciliation may follow.

 rule: design deploys so rollback needs ZERO data surgery.
 if reverting requires migration scripts, the deploy wasn't safe.

Making It Boring

 □ AUTOMATED triggers: gate breach → auto-rollback (with
   alert), no human latency in the loop (own lesson)
 □ REHEARSED quarterly: actually roll back in staging AND
   prod game days; measure wall-clock; fix what's slow
 □ LAST-KNOWN-GOOD always deployable: artifact retention,
   config snapshots, infra state pinned per release
 □ ROLLBACK ≠ FORGET: post-incident review of WHY the bad
   build passed gates — fix the gate, not just the symptom
 □ DRILL the data-path too: revert with recent v2-written
   data present; verify v1 handles it (tolerant readers proof)

Strategy	Rollback speed	Data caveat
canary weights	seconds	none usually
blue-green flip	seconds	shared-schema care
k8s rolling undo	minutes	probe-warm again
rebuild previous	tens of minutes	plan artifacts ahead

Interview Framing

“Bad deploy hits production — walk me through recovery” scored shape: time-budget framing (damage window = detect+decide+revert), strategy-specific mechanics with speeds, STATEFUL caveats led by schema-compatibility requirement, automation+drill disciplines named, post-incident gate-fix loop included. Rollback questions test whether safety was designed BEFORE the incident — rehearsed speed vs improvised heroics is the tell.