Retries

The Case for Retries

Transient failures are the majority: lost packets, restarted instances, momentary overload. A single retry converts many user-visible errors into invisible self-heals:

 per-call failure prob 0.1%:
   no retry   → 0.1% of requests fail
   1 retry    → 0.0001% fail (needs both attempts to fail)
   
 retries are the cheapest reliability multiplier available —
 WHEN applied to the right errors with the right policy

What Deserves a Retry

 RETRYABLE                          NOT RETRYABLE
 timeouts (maybe — state unknown!)  4xx validation/auth errors
 503 overloaded                     business rejections (insufficient funds)
 502/504 gateway errors             deterministic 5xx (bug will recur)
 connection reset/refused           non-idempotent ops without keys
 queue publish NACK
 
 decision rule: "same request, sent again, might succeed?"
 yes → retryable; no → fail fast to caller

The timeout row carries an asterisk that owns whole lessons: retrying non-idempotent POSTs without idempotency keys risks duplicate charges.

The Amplification Trap

 dependency degrades → latency spikes → your timeouts fire
 every client retries ×3 → traffic MULTIPLIES at exactly the
 moment the server is weakest

 healthy:    1000 rps
 degraded:   1000 rps real + 3000 rps retries = 4000 rps load
             → server falls further behind → MORE timeouts
             → more retries... retry storm = death spiral

 retries convert partial degradation into total collapse.
 this is why retry POLICY is safety-critical, not boilerplate

A Safe Policy

 - MAX 2–3 attempts total (not infinite)
 - EXPONENTIAL BACKOFF between tries (next lesson)
   + JITTER to break synchronization (next lesson)
 - budget-aware: never retry if deadline already blown
 - only on retryable classes (see table)
 - CIRCUIT BREAKER upstream of retries (own lesson):
     stop hammering when success rate collapses
 - retry budget/cap: e.g., retries ≤ 10% of baseline traffic,
   hard-capping storm multiplication

Where Retries Live Matters

 ✗ EVERY layer retries independently:
   client×3 × gateway×3 × service×3 = up to 27 backend hits
   
 ✓ retry at ONE designated layer (or coordinate budgets):
   edge retries for clients, mesh sidecar for internal hops,
   applications trust the layer below

 uncoordinated layers multiply silently — audit who retries what

Idempotency Prerequisite

Operation	Retry safe?
GET	Yes
PUT / DELETE	Yes
POST with idempotency key	Yes
Bare POST	NO — duplicates possible

Non-idempotent writes either gain keys or gain explicit “no-retry” policies. There is no third option.

Interview Framing

“Your downstream fails 2% of calls — fix it.” Scored arc: classify errors, retry-with-backoff-and-jitter for transients, cap attempts, mention the amplification trap and circuit breaker as the guardrail. The seniority signal is leading with the retry-storm risk before selling the benefit.