Idempotent Consumers

Why Every Consumer Needs This

 at-least-once is the industry default → duplicates are NORMAL:

 - producer retries after ambiguous timeout
 - broker redelivery after visibility timeout / rebalance  
 - consumer crash between process and commit

 your consumer WILL see the same event twice.
 the only question: does it survive that?

 IDEMPOTENT: f(f(x)) = f(x)
 processing an event N times = processing it once.

 non-idempotent consumers + at-least-once = silent corruption.

The Pattern Library

 1. DEDUPE TABLE (the workhorse):
    BEGIN;
      INSERT INTO processed_events(id) VALUES (:msg_id); -- PK!
      UPDATE inventory SET stock = stock - 1 WHERE sku = :sku;
    COMMIT;
    duplicate → PK violation → skip. atomic with the effect ✓

 2. NATURAL KEY UPSERT:
    INSERT ... ON CONFLICT (event_id) DO NOTHING
    or full upsert: same event → same final state, no drift

 3. CONDITIONAL VERSIONED WRITE:
    UPDATE accounts 
    SET balance = :new_bal, version = version + 1
    WHERE id = :id AND version = :expected_version;
    replay hits old version → 0 rows → correctly ignored ✓

 4. SET-SEMANTIC OPERATIONS:
    "set user status=active" — applying twice changes nothing.
    design operations as STATE TARGETS not DELTAS where possible:
    delta ops (stock-1) need dedupe; target ops often don't!

Choosing the Dedup Window

 dedupe table needs TTL/cleanup policy:

 window must EXCEED maximum redelivery horizon:
   broker retention ≥ dedupe window > worst retry span
 
 too short: duplicate arrives after cleanup → processed twice ✗
 too long:  table bloats (partition it by day; drop old)

 scale note: dedupe table itself needs scaling strategy —
 redis with TTL for high-rate/low-stakes flows,
 transactional SQL for money-class flows. match stakes to store.

Stakes	Store	Pattern
Analytics counters	Redis SETNX + TTL	cheap dedupe
Business state	SQL txn dedupe table	atomic-with-effect
Payments	versioned CAS + ledger	belt+suspenders
Notifications	natural keys	send-once registry

The Delta Trap

 subtle bug class worth naming:

 event: {order_id: 42, delta_qty: +5}

 naive: stock += 5        ← NOT idempotent!
 fixed: dedupe-table guard around it ✓
 better: carry ABSOLUTE values when you can:
         {qty_total: 15}   ← idempotent by construction

 producers who can emit absolute state make consumers
 trivially safe. prefer it whenever freshness allows.
 (CDC events do exactly this — row images, not mutations.)

Interview Framing

“Your queue delivers at-least-once — walk me through safe consumption” scored answer: name the duplicate SOURCES briefly, then patterns in stakes-order (dedupe-table-in-same-txn as flagship), the delta-vs-target operation distinction (the senior insight), window sizing vs redelivery horizon, and the closing equation: “at-least-once delivery + idempotent consumers ≈ exactly-once effects.” If interviews had a single required messaging answer, this is it.