Hot Partitions

How Partitions Get Hot

 three recurring causes:

 1. CELEBRITY KEY:    one user/post/product out-draws thousands
                      hash(key) → ONE shard regardless of cluster size

 2. TEMPORAL PILEUP:  range-sharded by timestamp → all of TODAY's
                      writes land on today's shard

 3. SKEWED ENTITIES:  city=nyc, tenant=enterprise-A —
                      natural data imbalance baked into keys

 result: one node at 95%, fleet at 12% average.
 alerts fire on the HOT node; capacity math lies.

Detection Before Incidents

 per-partition metrics (never fleet averages alone):

 - requests/sec BY SHARD      alert on max/min ratio > 3-5x
 - partition SIZE distribution (Cassandra: nodetool tablehistograms)
 - per-key access sampling    (Redis LFU; app-level top-K sketches)

 heavy-hitter detection algorithms (count-min sketch) at the
 service layer identify hot keys in real time cheaply.

Mitigation 1: Splitting (Range Systems)

 overloaded range splits into two:

 [1M–2M) too hot → [1M–1.5M) + [1.5M–2M)
 half migrates to a new node automatically (Spanner-class systems)

 works when: load is spread across MANY keys in the range
 fails when: ONE key IS the load (splitting a celebrity's range
             still leaves the key itself on exactly one shard)

Mitigation 2: Salting / Bucketing

 append a random suffix to spread ONE logical key:

 write: shard_key = hash(user_id + ":" + random(0..15))
        → celebrity's writes spread over 16 shards

 read:  fan to all 16 buckets, merge/sort results
 counter reads: SUM across buckets

 trade-offs:
 + writes scale ×buckets
 − reads multiply by bucket count  
 − global ordering within the entity is lost
 
 bucket count tuned to expected hotness (16 typical, 64 for mega).

 for TIME ranges: salt WITHIN each time bucket:
   PK((day, salt_bucket), ...) → today spreads over buckets,
   day queries scan buckets in parallel. standard TSDB pattern.

Mitigation 3: Tiering the Hot Entity

 move the celebrity OFF the general path entirely:

 - cache their reads aggressively (L1+redis; hot = read-heavy!)
 - queue their writes (fan interactions through buffer)
 - dedicated storage pool for known-mega entities
   (routing table exception: celebrity_id → special shard group)

 platform reality: your top 10 entities need bespoke handling;
 design the escape hatch BEFORE the first viral moment,
 since retrofitting under fire is how cascades start.

The Playbook Summary

Cause	First response	Structural fix
Celebrity key	Cache hard	Salting/bucketing
Time pileup	Pre-split next bucket	Salted compound keys
Natural skew	Rebalance vnodes	Key redesign + tiering

Interview Framing

“Your sharded DB has one node at 90% CPU” scored diagnosis sequence: identify WHICH cause via per-key metrics, apply matched mitigation (not generic “add nodes”!), state read/write tradeoff of salting with bucket-count arithmetic, mention tiering as the celebrity endgame. Diagnose-then-treat ordering is what’s being tested — pattern-matching to “add shards” fails.