Cross-Shard Transactions

When One Transaction Spans Shards

 debit user A (shard-1), credit user B (shard-3):

 single-node ACID can't help — two databases now.

 without coordination:
   debit commits, credit fails → MONEY VANISHES
 
 distributed transaction options exist on a cost spectrum;
 and the BEST option is usually redesigning so it isn't needed.

Option 1: Two-Phase Commit (2PC)

 PHASE 1 (prepare): coordinator asks all shards "can you commit?"
                    each locks resources, votes yes/no
 PHASE 2 (commit):  ALL yes → commit everywhere
                    ANY no  → abort everywhere

 [coord] ──prepare──► s1 ✓   s2 ✓   s3 ✓
 [coord] ──commit ──► s1     s2     s3      atomic ✓

 THE COSTS (why everyone avoids it):
 - LOCKS HELD during both phases → throughput collapses at scale
 - BLOCKING: coordinator crash mid-phase leaves shards locked
   (they must wait/resolve — recovery protocols are complex)
 - latency = slowest participant × 2 phases
 - availability hit: ANY shard down = transaction impossible

 usable for LOW-volume coordination; never hot paths.

Option 2: Sagas (Compensation)

 break the atomic txn into LOCAL steps + compensating actions:

 TransferSaga:
   1. debit A          (local commit ✓)
   2. credit B         (local commit ✓)
      failure? → 3. compensate: credit A back

 state machine with retries + compensations:
 [debit]──►[credit]──►done
    │         │
    ▼         ▼
 refund-A  (retry credit)

 properties:
 + no locks across shards; high throughput  
 − NOT atomic in between (money "in flight" — visible!)
 − compensation logic is real code with real bugs
 − needs idempotency everywhere (own lesson)

 standard for microservices; acceptable UX for most domains.

Option 3: Design It Away (the senior answer)

 most cross-shard transactions are KEY-DESIGN failures:

 ✗ users sharded by id → transfers always cross-shard
 ✓ CO-LOCATE related entities:
    shard by user_id → user's wallet+orders+cards same shard
    → local ACID transactions!

 transfer A→B still crosses... unless:
 - LEDGER pattern: append-only entries, one per account,
   written locally + reconciled asynchronously (banks do this)
 - queue-mediated: events to a per-account processor
   serializing each account's mutations

 question to ask BEFORE choosing 2PC/sagas:
 "what key change or data model makes this LOCAL?"

Decision Table

Situation	Choice
Entities co-locatable	Redesign keys → local txns
Money movement, high volume	Ledger + async reconciliation
Multi-service workflow	Saga with compensations
Rare, low-volume coordination	2PC tolerable
Truly need strict atomicity at scale	Distributed SQL (Spanner-class)

Interview Framing

Cross-shard transaction probes separate architects from feature-developers. Scored order: FIRST propose key/co-location redesign (“can we make it local?”), THEN saga with compensation example and its visibility caveat, name 2PC only to dismiss it with lock/blocking costs, mention ledger pattern for money. That priority ordering — redesign before protocol — is precisely what’s being evaluated.