Exactly-Once Delivery

The Claim vs The Physics

 EXACTLY-ONCE: every message processed exactly once.
 never lost, never duplicated.

 the two-generals problem says perfect exactly-once delivery
 across unreliable networks is impossible in general:
 the final ack can always be lost, forcing either
 a retry (duplicate) or an acceptance (possible loss).

 so what do systems SELLING exactly-once actually provide?

What Kafka Transactions Actually Give You

 kafka's exactly-once semantics (EOS) scope:

 ✓ consume-transform-produce PIPELINES atomic:
   input offsets + output messages commit together
   crash → whole unit replays or completes, never half

 ✓ idempotent producer: retries don't duplicate within session

 what it does NOT cover:
 ✗ your SIDE EFFECTS outside kafka (the email already sent!)
 ✗ arbitrary external database commits (unless transactional
   integration exists)
 ✗ consumer-side non-kafka state

 [kafka]──EOS──►[kafka]  = exactly-once ✓
 [kafka]───────►[send email] = at-least-once + hope

Exactly-Once EFFECTS: The Achievable Goal

 reframe: you can't control DELIVERY exactly-once end-to-end,
 but you can make PROCESSING have exactly-once effects:

 techniques composing to the goal:

 1. DEDUPE TABLE in same txn as effect:
      BEGIN; 
        INSERT INTO processed(msg_id);   ← unique constraint!
        UPDATE accounts SET ...;
      COMMIT;
    duplicate arrives → constraint violation → skip. effect once.

 2. NATURAL IDEMPOTENCY: design operations as idempotent upserts
    (set balance=X given version — same result applied twice)

 3. TRANSACTIONAL OUTBOX for outgoing effects:
    side effects recorded atomically, delivered by relay
    (relay retries safely against idempotent receivers)

 this stack delivers what businesses actually need:
 "this charge happens once" — regardless of delivery chaos.

The Cost Ledger of True EOS

Cost	Magnitude
Throughput	transactions cut kafka-class throughput significantly
Latency	commit coordination adds ms per batch
Complexity	transactional APIs fence/epoch management
Scope discipline	every new sink needs integration thought

 adopt when: money movement, inventory decrements, billing events
 skip when: analytics events (at-least-once + dedupe downstream
            is cheaper), notifications (idempotent by design anyway)

Interview Framing

Exactly-once questions are trapdoors for the unprepared. Scored position: state the impossibility result plainly (“perfect EOS delivery is theoretically impossible — ack loss forces retry-or-loss”), explain what framework EOS scopes (pipeline atomicity), then pivot to exactly-once EFFECTS via same-transaction deduping as the practical goal. Candidates who parrot “Kafka does exactly-once” without scoping get dismantled; candidates who reframe delivery-vs-effects impress.