Zero-Downtime Deployments

Zero Downtime Is a System Property

 no single trick delivers it; it's a CONTRACT between layers:

 [LB]──health-aware──►[instances: readiness gates]
        │
   graceful drain on terminate
        │
   backward-compatible code/schema during transitions
        │
   sessions/state that survive instance death

 break any layer and users see blips, errors, lost carts —
 usually only during deploys, the hardest class of bug to repro.

The Layer Checklist

 L1 TRAFFIC: LB/routes never point at unready instances;
    health checks with meaningful depth (not just TCP)
 L2 INSTANCES:
    □ readiness probe = genuinely able to serve (deps warmed)
    □ preStop/drain: stop accepting NEW requests,
      finish IN-FLIGHT ones (grace period sized to p99!)
    □ connection pools closed politely, not RST-storms
 L3 COMPATIBILITY: N/N+1 API tolerance + expand-contract
    schemas (the whole backward-compatibility discipline)
 L4 STATE: externalized session stores / stateless design —
    an in-memory cart dies with its pod mid-roll
 L5 CLIENTS: retry-on-idempotent + jitter so transient
    edge cases (if any leak through) self-heal silently

The Classic Blunders

Symptom	Root cause
error burst at roll START	readiness lies (cold caches)
errors at roll END	missing drain; in-flight killed
sporadic 502s from LB	health-check interval too slow for pod churn
user carts vanish	state in memory
one endpoint breaks only during deploys	N/N+1 violation hiding there
DB connection storm post-roll	every new pod opens full pool instantly

 the LAST-POINT failure is sneaky and common:
 stagger pod starts (or warm-up windows) so fresh pods ramp
 connections gradually instead of stampeding shared stores.

Verifying Zero-Downtime Claims

 claims require experiments:

 □ DEPLOY-STORM DRILL: trigger rolls while synthetic real-
   shaped load runs; assert ZERO client-visible errors
   (not "few" — zero, or enumerate why)
 □ KILL-CHAOS variant: random pod deletes under load —
   same assertion; drains and retries must absorb it
 □ measure the USER side (client metrics), not server logs —
   servers lie to themselves about their own blips
 □ include SCHEMA-migration deploys in drills: the riskiest
   windows are exactly the ones teams forget to rehearse

 when the drill fails: trace WHICH layer leaked
 (traffic? instance lifecycle? compatibility? state?) —
 the checklist maps symptoms to culprits directly.

Interview Framing

“Prove this service deploys without user impact” scored shape: five-layer contract enumerated, classic-blunder table condensed from experience, drill-based verification with zero-error assertion emphasized, staggered-warmup subtlety included. Zero-downtime questions separate people who configure k8s defaults from people who’ve chased deploy-window bugs at 3am — the drain-and-warm details are where that shows.