Retries

The Instinct and Its Trap

 call fails → try again. obviously right... sometimes:

 RETRY-WORTHY:                    NEVER-RETRY:
 - network timeout (ambiguous)    - 400 validation error
 - 503 overloaded briefly         - 401/403 auth failure
 - 502/504 gateway hiccups        - 422 business rejection
 - connection reset               - deterministic logic errors

 blanket retries on permanent failures = guaranteed waste.
 classify FIRST, retry SECOND.

 THE BIGGER TRAP — retry AMPLIFICATION:

   1 caller × 3 attempts = 3× load
   layered services each retrying 3×:
   3 × 3 = 9× load against an already-struggling dependency
   
   [api]──retry×3──►[orders]──retry×3──►[payments]
   
   payments at 50% capacity receives 9× normal requests.
   retries turned degradation into death spiral.

The Rules That Make Retries Safe

 □ BUDGET-CAPPED ATTEMPTS: max 2–3 total, never infinite
 □ EXPONENTIAL BACKOFF + JITTER: 100ms → 200 → 400 ±random
   (synchronized retries = self-DDoS; own lessons)
 □ IDEMPOTENCY REQUIRED for anything non-read:
   did attempt #1 actually succeed before the timeout?
   duplicate charges/emails/orders await the careless.
   (idempotency keys or safe-operation design)
 □ TOTAL-TIME ENVELOPE: attempts fit inside caller deadline
     budget 800ms ≥ 3 tries × (200ms + backoff)
 □ HEDGING ≠ RETRYING: hedge = send duplicate EARLY while
   first still in-flight (tail-latency trick); costs 2× load;
   use only on idempotent reads with tight budgets.

Retry Only Where It Pays

 WHERE to place retries in a stack — exactly ONE layer:

 ✗ client retries + api retries + service retries
   = multiplication again.

 ✓ pick the layer closest to the FAILURE CAUSE:
   transient network blip between svc A→B? retry IN A→B client.
   whole-endpoint failure under load? shed/circuit-break instead,
   DON'T retry — adding load is the opposite of help.

 rule of thumb: retry TRANSIENT faults; CIRCUIT-BREAK
 sustained ones; SHED overload. three tools, three diseases.

Reading the Signals

Symptom	Likely retry pathology
Load spikes AFTER incident starts	amplification cascade
Duplicate side effects reported	missing idempotency
p99 explodes during partial brownouts	retry storms
Recovery delayed by minutes after fix	queued retry backlog draining

 instrument: attempts-per-request histogram, retry-rate per
 dependency, duplicate-detection hits. you can't fix
 what the dashboards don't show.

Interview Framing

“Payment gateway intermittently times out” scored answer: classify (transient → retryable), then IMMEDIATELY pair with idempotency (“retries require idempotency keys here”), show backoff+jitter and budget arithmetic, cap attempts, mention circuit-breaker if it degrades further, warn about layered-retry multiplication explicitly. The idempotency pairing is what separates veterans from tutorial-followers.