Message Retries

The Consumption Failure Model

 consumer processing fails two ways:

 TRANSIENT:   downstream DB blip, timeout, network flap
              → retry will succeed later ✓
 PERMANENT:   malformed message, missing referenced entity,
              business-rule rejection → retry NEVER helps ✗

 the entire discipline is distinguishing these,
 because retrying permanent failures forever:
 - clogs the queue behind a poison head
 - burns compute on guaranteed failure
 - delays everything else in the partition/queue

The Retry Lifecycle

 attempt fails → broker redelivers:

 t+0     attempt 1  fails (transient?)
 t+30s   attempt 2  ← backoff grows each round
 t+2m    attempt 3
 t+10m   attempt 4
 ...after N attempts → DEAD LETTER QUEUE (own lesson)

 mechanisms by platform:
 - SQS:      visibility timeout + receive-count limit
 - rabbit:   TTL + dead-letter exchanges, retry queues with TTLs
 - kafka:    retry TOPICS per backoff tier + offset tracking
 
 all implement the same state machine; know yours concretely.

Backoff for Consumers

 same exponential+jitter math as HTTP retries:

 delay = base × 2^attempt ± jitter

 why jitter matters MORE here:
 consumers process partitions serially — synchronized retries
 of many messages = self-DDoS on your own downstream.
 spread them.

 cap total attempts AND total time:
   max_attempts = 5
   max_age      = 24h   (stale messages often meaningless anyway —
                         "send password reset" from yesterday?)

Classifying Failures in Code

 make retry decisions EXPLICIT, not accidental:

 catch (error):
   if error.type == VALIDATION:      → DLQ immediately (no retry)
   if error.type == NOT_FOUND and age > 5m: → DLQ (dependency gone)
   if error.type == TIMEOUT/CONFLICT: → retryable ✓
   if error.type == RATE_LIMITED:    → retry with LONGER backoff
   else:                              → default retry path, alert

 anti-pattern: blanket catch-all retries.
 validation errors retried 5× over an hour = pure waste +
 delayed DLQ arrival + confused dashboards.

Retry Storms and Their Dampers

 downstream outage + N pending messages × retries =
 multiplication against a dying dependency:

 dampers, in combination:
 □ CIRCUIT BREAKER before the failing downstream:
   open circuit → pause consumption entirely (backpressure)
   instead of generating guaranteed-failing attempts
 □ global retry budget: pause topic when >X% failing
 □ consumer-side rate limiting toward sick dependencies
 □ alert on failure RATE not count (volume scales with traffic)

Interview Framing

“Consumer keeps failing — walk me through handling” scored shape: transient-vs-permanent classification FIRST, backoff-with-jitter mechanics, explicit exception-type routing table, DLQ as the terminal state (teaser), and circuit-breaker coupling to stop retry storms at the source. The classification-first instinct is what separates handlers from hope.