Transactional Outbox

The Problem It Solves

 naive: save to DB, then publish to broker.

 db.save(order); broker.publish(OrderPlaced);
        │                  │
   committed ✓      CRASH — event never sent.
   
 downstream never learns. silent inconsistency.
 the reverse order creates phantom events instead.

 root cause: two systems, no shared transaction.

The Pattern

 STEP 1 — atomic write of business data + event:

 BEGIN;
   INSERT INTO orders (id, ...) VALUES (...);
   INSERT INTO outbox (id, aggregate_id, type, payload, created_at)
     VALUES (uuid, order.id, 'OrderPlaced', '{"..."}', now());
 COMMIT;    ← both rows or neither. hole closed ✓

 STEP 2 — separate relay ships outbox → broker:

 loop:
   rows = SELECT * FROM outbox 
          WHERE published = false ORDER BY created_at LIMIT 100;
   for row: broker.publish(row.payload)
            UPDATE outbox SET published = true WHERE id = row.id;

 [app txn]──► [outbox table] ──relay──► [kafka/rabbit]
                    │           │ fails? retries safely ✓
                    └─ same DB ─┘

Relay Design Details

 the relay is small but has sharp edges:

 □ AT-LEAST-ONCE by nature: crash after publish,
   before marking → duplicate publish on retry.
   CONSUMERS MUST BE IDEMPOTENT (they always should be).

 □ COMPETING RELAYS need coordination:
   single relay (simple; SPOF mitigated by k8s restart) or
   row-locking / partitioned claims for parallel relays.

 □ POLLING vs CDC delivery:
   poll: simple, adds latency (poll interval), DB load
   CDC:  tail the WAL/log — near-real-time, less app-visible state
         (debezium reads outbox table directly!)

 □ CLEANUP: published rows deleted/archived on schedule;
   unbounded outbox tables become their own incident.

 □ ORDERING: emit in created_at/id order per aggregate;
   sequence numbers help consumers reorder-guard anyway.

Why Not Alternatives

Alternative	Why it loses
2PC across DB+broker	locks, latency, spotty broker support
Publish-then-save	phantom events on save-failure
Save-then-publish-with-retry	can’t know if publish “took”; still racy
Async-after-response	process death between steps = lost forever

 outbox wins by REDUCING the problem to local transactions
 plus an at-least-once pipe — both solved problems.

Interview Framing

“Order service must reliably notify inventory and email” scored shape: name dual-write as the flaw FIRST, implement outbox (same-txn insert + relay loop), address relay duplicates → idempotent consumers explicitly, offer CDC-vs-polling variant, cleanup noted. Drawing the two-phase flow with the failure point crossed out (“crash here → safe”) communicates the pattern faster than words — diagram it.