Poison Messages

What Makes a Message Poisonous

 POISON MESSAGE: reliably crashes/fails every consumer that
 touches it. not transient — DETERMINISTIC failure.

 classic causes:
 - malformed payload (schema drift: producer updated, consumer didn't)
 - null where code assumes value (deserializer NPE)
 - size explosion (100MB blob in a 1MB-memory consumer → OOM)
 - infinite loop trigger (malicious or accidental)
 - dependency expectation violated (event for deleted entity,
   retried forever by hopeful retry logic)

 the danger isn't one failure — it's the LOOP:
 crash → redelivery → crash → redelivery...

The Stall Cascade

 serial consumption + poison head = partition frozen:

 [poison][msg2][msg3][msg4]...
    ▲
    consumer dies here, restarts, dies here again

 with kafka: offset never advances past it.
 EVERYTHING behind waits. lag climbs. alerts fire.
 worst case: consumer group's restart loop burns compute
 and hides OTHER failures behind the noise.

Defense Layers

 1. VALIDATE AT INGEST (best):
    schema registry + validation at produce time;
    poison never enters the queue at all ✓

 2. FAIL FAST, COUNT, QUARANTINE:
    try process(msg):
      catch fatal:
        attempts += 1
        if attempts >= LIMIT: → DLQ immediately
        else: nack-with-retry
    
    the LIMIT is everything. without it: infinite loop.

 3. ISOLATE PROCESSING:
    per-message subprocess/timeout so OOM/hangs don't kill worker
    (heavier; reserved for untrusted payloads)

 4. CONSUMER HEALTH CIRCUIT:
    same message failing K times in M minutes →
    auto-skip to DLQ + alert, keep partition flowing

Schema Drift: The #1 Poison Factory

 producer deploys new field/consumers don't know it:

 {amount: 2000}          ← old consumers fine
 {amount: 2000, tax: 90} ← new producer; old consumer's strict
                            parser throws on unknown field!

 prevention:
 □ backward-compatible schemas ONLY (additive, defaulted)
 □ contract tests in CI for both directions  
 □ canary consumers on new schema versions
 □ tolerant readers: ignore unknown fields (protobuf default;
   JSON parsers often configurable)

 this is why schema-registry lessons exist — poisoning
 is usually an EVOLUTION accident, not bad luck.

Triage When Found Live

 production incident shape:

 1. IDENTIFY: which offset/id keeps failing (logs show loop)
 2. CONTAIN: skip/quarantine it — restore flow FIRST
    (kafka: manual commit past it after copying to side store)
 3. UNDERSTAND: reproduce offline with quarantined copy
 4. FIX: parser hardening / backfill correction
 5. REPLAY via proper re-drive (DLQ lesson)
 order matters: flow restoration before root-causing.

Interview Framing

“A single malformed event froze your Kafka consumer group” scored response: name the stall mechanics (offset stuck, cascade), immediate containment-before-diagnosis ordering, the attempt-limit-to-DLQ defense as systemic fix, and schema-drift prevention (registry + tolerant readers) as root cause. The contain-first instinct marks operational experience.