Resharding

Why Resharding Is Inevitable

 sharding decisions made at scale X face scale 10X:

 - more shards needed (capacity)
 - different partition key (query patterns evolved)
 - rebalancing after hotspots/uneven growth
 
 resharding = moving live data to a new layout with ZERO
 downtime. it's one of the hardest routine operations in
 distributed systems — plan it as a PROJECT, not a migration script.

The Four-Phase Lifecycle

 PHASE 1 — EXPAND (dual-write):
   deploy writes to OLD and NEW layouts simultaneously
   (transactional outbox/CDC makes this reliable)
   new layout fills going forward

 PHASE 2 — BACKFILL:
   copy historical rows old→new in batches:
   SELECT id > :cursor ORDER BY id LIMIT 10000;
   throttle to protect production; track progress per range
   
 PHASE 3 — VERIFY + SWITCH READS:
   checksum/count comparison per range; fix discrepancies
   then migrate reads INCREMENTALLY: 1% → 10% → 50% → 100%
   comparing results shadow-mode first if paranoid (be paranoid)

 PHASE 4 — CLEANUP:
   stop dual-write, retire old layout AFTER observation window
   (weeks later — forgotten batch jobs exist)

The Dual-Write Trap (and the fix)

 naive app-code dual-write fails:

 write A(old) ✓ , write B(new) ✗ crash → DIVERGENCE forever.

 correct mechanisms:
 TRANSACTIONAL OUTBOX: business txn + change-event committed
   atomically; relay applies to both layouts with retry.
 CDC: read the primary's log; derive ALL destinations from it.
   single source of truth for change order — no divergence possible.

 CDC-based is the production default. own lessons cover both deeply.

Online Migration Mechanics

 keeping reads consistent mid-migration:

 version-stamped routing:
   each row knows its home (old/new) during transition
   router checks flag → directs accordingly

 incremental cutover by KEY RANGE:
   users A-F on new shards, G-Z still old → per-range flip
   blast radius of any bug = one range = rollback is trivial

 rollback discipline at EVERY phase:
   phase N must be reversible without data loss;
   if not reversible, it's two phases.

Resharding Runbook Essentials

Item	Why
Progress dashboard	Backfills stall silently
Throttle controls	Protect prod from your backfill
Checksum verification queries	Trust nothing; verify everything
Per-phase rollback scripts	Panic-proof the operation
Freeze windows for schema edges	Some ALTERs can’t race migration

 timeline reality: weeks not hours.
 dual-write week-1, backfill weeks 1-3 (throttled),
 verify+shadow days, gradual cutover week-4,
 cleanup month-2. schedule accordingly.

Interview Framing

“How would you move from 4 to 16 shards live?” scored arc: four-phase lifecycle named, CDC-vs-dual-write distinction made (with divergence example), incremental per-range cutover for bounded blast radius, verification-before-trust emphasis, cleanup-after-observation-window patience. Candidates proposing “dump and restore” fail instantly; the lifecycle IS the answer.