Dead Letter Queues

The Problem They Solve

 a message that always fails sits at queue HEAD:

 [poison msg] ──► retry ──► fail ──► retry... forever
      │
      ▼
 everything BEHIND it waits. partition stalls.
 one bad message = entire pipeline outage.

 DLQ: after N failed attempts, MOVE IT ASIDE.

 [queue] ──N fails──► [DLQ] ──► humans/repair tooling
    ▲                                    │
    └──────── re-drive after fix ◄───────┘

Anatomy of a Good DLQ Entry

 the moved message must carry its AUTOPSY DATA:

 original payload           ← untouched
 failure reason             ← exception type + message
 stack trace / context      
 attempt count              ← how many times it died
 first/last failure time    
 source topic + partition + offset  ← provenance!

 platforms vary:
 - rabbit:   x-death headers accumulate automatically
 - sqs:      redrive moves message; metadata in attrs
 - kafka:    often DIY: publish envelope {original, error, meta}
 
 without this data, DLQ = graveyard, not diagnostic tool.

Operating the DLQ

 DLQ is a WORKFLOW, not a trash can:

 □ ALERT on depth > threshold — every DLQ item is a real
   failure someone must understand (not noise to ignore!)
 □ TRIAGE cadence: classify each entry:
     BUG?        → fix code → RE-DRIVE messages
     DATA ISSUE? → repair payload or skip with record
     PERMANENT?  → document why, archive, drop
 □ RE-DRIVE carefully: replay into source topic in ORDER,
   throttled — a bulk redrive can stampede consumers
 □ TTL/capacity on the DLQ itself (it's a queue too!)

The Re-Drive Pattern

 fix deployed → replay failures:

 for msg in dlq.batch(100):
     if msg.created_at < fix_deploy_time:   ← only old ones!
         republish(msg.original)
         ack-delete(msg)
     else:
         leave  ← new failures might be a NEW bug

 guards:
 - re-drive is IDEMPOTENT-safe by definition needed
   (some may have partially succeeded before)
 - monitor success rate during replay; abort if failing again
 - cap replay rate (it's traffic like any other)

Design Decisions

Decision	Options	Guidance
Scope	one DLQ vs per-topic	Per-source-topic (triage clarity)
Threshold	attempts N	3–5 typical
Retention	DLQ message age	≥ incident-resolution horizon
Ordering	preserve?	Usually no — these are exceptions

 anti-patterns:
 ✗ single global DLQ for 50 topics (untriageable soup)
 ✗ DLQ monitored by nobody (silent data loss with extra steps)
 ✗ auto-redrive loops ("retry DLQ forever" = no DLQ)

Interview Framing

“Messages keep failing in your pipeline — what happens to them?” scored answer: poison-head stall as motivation, move-after-N mechanics, DLQ ENTRY CONTENTS (provenance + error context) as the differentiator, operational loop (alert-triage-fix-redrive) with ordered throttled replay. Mentioning that an unmonitored DLQ is just organized data loss shows production maturity.