The Dual-Write Problem

The Pattern That Always Fails

 anywhere code touches TWO systems without a shared transaction:

 db.save(X);  cache.set(key);          ← cache invalidation!
 db.save(X);  broker.publish(Event);   ← event publishing!
 db.save(X);  search.index(doc);       ← search sync!

 ALL are dual-writes. ALL have the same hole:

 ┌─────────┐     ┌─────────┐
 │ write A │     │ write B │
 └─────────┘     └─────────┘
      ▲ crash/fail between = inconsistency forever

 no error surfaces. no retry knows. state DRIFTS silently —
 discovered days later as "weird" bugs nobody can reproduce.

The Failure Catalog

 enumerate the gaps precisely:

 order of writes doesn't matter — both orders fail:

 save→publish, crash between:  fact exists, world uninformed
 publish→save, save fails:     phantom event, no fact behind it

 subtler variants:
 - network timeout on B (did it happen? unknown!)
 - partial success (B committed in cluster but ack lost)
 - process killed by OOM/deploy mid-sequence
 - exception AFTER A committed, caught too broadly

 every async integration you've written has this bug
 unless you applied one of the fixes below.

Fix Inventory

Fix	Mechanism	Best for
Transactional outbox	event-as-row in same txn	events to broker
CDC from source	log IS the event stream	table-shaped facts
Write-through + TTL	cache heals via expiry	cache staleness
Reconciliation job	periodic diff & repair	search indexes
Distributed txn (2PC)	true atomicity	rare, costly cases

 outbox/CDC dominate for messaging;
 reconciliation is the safety net for systems lacking logs.
 caches get TTL grace. pick per pair-of-systems.

Detection When Prevention Failed

 drift happens anyway — instrument for it:

 □ RECONCILIATION CHECKS: sampled/periodic comparison
   (count mismatches, checksum windows) → alert on divergence
 □ SEQUENCE GAPS: consumers detect missing sequence numbers
   per key → flag lost events
 □ AGE LAG METRICS: oldest un-synced row per target system
 □ AUDIT TRAILS: enough recorded intent to REPLAY repairs

 the goal isn't zero drift forever (unrealistic);
 it's drift DETECTED IN MINUTES with replayable repair,
 instead of discovered by support tickets in weeks.

Interview Framing

Design reviews score this silently: any diagram where a service writes DB and emits events gets asked “what if it crashes between?” Candidates who answer “outbox” immediately pass; candidates who never considered it reveal untested designs. Internalize the reflex: two writes, no shared transaction = find the fix BEFORE shipping. This lesson is the diagnosis; the outbox lesson is the cure.