Hot Keys

The Problem

 partitioning spreads load by KEY. skew breaks it:

 celebrity account: 2M rps on ONE key
 hash(key) → exactly ONE cache node / db shard

 that node: on fire, timing out
 other 11 nodes: 3% utilized, watching
 
 no amount of horizontal capacity fixes a single-key hotspot —
 the key IS the unit of distribution.

Detection

 metrics that reveal hotspots:
 - per-node request rate variance (one node ≠ fleet average)
 - redis: --hotkeys, or LFU sampling for top accessed keys
 - shard-level latency divergence in traces

 alert on SKEW (max/min node load ratio > 3),
 not just absolute load — hotspots appear at 30% average
 utilization and still cause incidents.

Mitigation 1: Key Replication

 store N copies under derived keys; read round-robin:

 write: set to post:77:c0 .. post:77:c9      (N replicas)
 read: pick random c_i → load spreads over 10 nodes

 trade-offs:
 + reads scale ×N
 − invalidation must hit all copies
 − memory ×N
 − consistency window between copies (fine for read-mostly)

 classic for viral content, config blobs, trending entities:
 data is read-heavy by definition if it's hot.

Mitigation 2: Local In-Process Cache Layering

 put L1 (per-pod memory) in front of shared Redis:

 [pods] ──L1: post:77 (TTL 1-5s)──► [redis] ──► [db]

 200 pods × own copy = aggregate read capacity scales
 with POD COUNT automatically. staleness bounded by tiny TTL.

 this is THE standard answer for extreme read hotspots:
 short-TTL local caching converts one hot key into
 thousands of independent micro-caches.

Mitigation 3: Split What Can Be Split

 sometimes the hot VALUE has structure:

 counter:  split into SLOTS ctr:77:{0..15}
           increment random slot; SUM for reads
           (or flush slots periodically to a merged total)

 list/feed: paginate/chunk across keys instead of one giant value

 geo heat:  NYC concert keys could carry city prefix routing
            to a pre-scaled dedicated pool

When It’s Writes, Not Reads

 write hotspots (one shard taking most WRITES):
 - can't replicate your way out (writes amplify!)
 - options: buffer writes (queue/write-behind), slot-split counters,
   or re-shard with a salted key: hash(key + bucket) where
   bucket = random(0..N) chosen per write — trading scanability
   for spread. heavy machinery; design keys to avoid needing it.

Interview Framing

“Taylor Swift joins your platform” is the meme-probe for hotspot handling. Scored arc: name single-key distribution as root cause, detect via skew alerts, then L1-with-short-TTL as primary fix (“scales with pod count”), key replication for cross-pod sharing, slot-splitting if it’s a counter. The insight being tested: you cannot fix a hot KEY with more NODES alone.