Rebalancing

Why Balance Drifts

 perfect balance is a moment, not a state:

 - uneven growth: shard-2's tenants grow 3x faster
 - new nodes join empty (everything else full)
 - hotspots emerge (celebrity entities)
 - node replacement: fresh disk must receive its share

 over months: [78%][41%][12%][66%] — capacity planning lies,
 the fullest node dictates the ceiling. rebalancing restores
 the premise that horizontal scaling was sold on.

What Rebalancing Moves

 depending on system architecture:

 VIRTUAL SHARDS/SLOTS:  move slot-ownership between nodes
                        (redis cluster: 16384 slots migrate)
 TOKEN RANGES:          split + transfer ranges 
                        (cassandra/spanner-class)
 VNODES:                reassign vnode→node mapping; data follows
 LEADERS/REPLICAS:      shuffle leadership roles for load
 
 virtualization granularity determines SMOOTHNESS:
 1024 virtual shards → tiny incremental moves;
 4 physical shards → each rebalance is a monster migration.

The Mechanics (safe movement)

 moving range [1M–1.5M) from A to B:

 1. COPY:    stream rows A→B (throttled); B indexes build
 2. CATCH UP: apply changes that occurred during copy (CDC/log)
 3. FLIP:    brief metadata switch — routing points to B
             (atomic directory update / consensus record)
 4. CLEANUP: A drops migrated rows AFTER confidence window

 invariants throughout:
 □ every key has EXACTLY one authoritative home at any instant
   (or dual-read resolution during transition)
 □ throttling protects foreground traffic (rebalancing is a guest)
 □ resumable: crash mid-move leaves recoverable state

When to Rebalance (automation with guardrails)

 TRIGGERS:
 - disk skew > threshold (max-min > 15-20%)
 - request-rate skew sustained (hot partition signal)
 - node added/removed (the obvious case)
 - scheduled drift review (monthly)

 GUARDRAILS against automation eating production:
 - max concurrent moves per cluster (usually 1!)
 - bandwidth caps on migration traffic
 - blackout during peak/business events
 - human approval above size thresholds
 
 auto-scaling ADDS nodes; auto-rebalancing FILLS them.
 both need both — capacity without distribution is theater.

Rebalancing vs Resharding

 related operations, different scopes:

 REBALANCING: same layout, better placement
              (move ranges/vnodes; hours; routine)
 RESHARDING:  different layout entirely  
              (new keys/count; weeks; project)

 good virtual-shard design makes most growth problems
 REBALANCING (boring) instead of RESHARDING (terrifying).
 this asymmetry is WHY virtualization-from-day-one matters.

Interview Framing

“Cluster is unevenly loaded after 6 months” scored answer: identify what moves (slots/ranges/vnodes), safe-movement sequence (copy-catch-flip-clean) with single-home invariant, throttle-and-guardrail automation stance, and the rebalance-vs-reshard distinction crediting early virtual-shard planning. The phrase “rebalancing is a guest in production” lands well because it’s true.