Distributed Cache Architecture

Why Distribute

 single cache node limits:
 - MEMORY:      8-64GB practical ceilings vs TB-scale hot sets
 - THROUGHPUT:  one NIC, one CPU (redis is mostly single-threaded!)
 - AVAILABILITY: node restart = total cold start

 distributed cache: keys spread across N nodes;
 capacity/throughput scale additively.

Topology: Sharding by Hash

 [app pods] compute slot = hash(key) % N
            └──► talk DIRECTLY to the owning node

 node-0: keys hash→0     ┐
 node-1: keys hash→1     ├─ each independent; no coordination
 node-2: keys hash→2     │   for reads/writes
 node-3: keys hash→3     ┘

 implementations:
 - Redis Cluster:    16384 hash slots, native routing/moves
 - Memcached fleet:  clients agree on the hash function
 - Cloud managed:    same concepts, elastic operation

Consistent Hashing Enters

 naive hash(key) % N breaks on resize:
   N: 4→5 → ~80% of keys remap → instant full-cache flush

 CONSISTENT HASHING (ring):
   nodes placed on a ring by hashed identity
   key owned by next node clockwise
   adding node k steals only ~1/N of key space

 [ring]  n0 ─ n1 ─ n2 ─ n3
   add n4 between n1,n2 → only that arc's keys move

 virtual nodes (100-200 per physical) smooth distribution
 and let beefier nodes take proportionally more arcs.

Client-Side vs Proxy Routing

Style	How	Trade-offs
Smart client	Library knows topology, routes direct	Lowest latency; client complexity
Proxy tier	Envoy/twemproxy in front	Simple clients; extra hop
Cluster-native	Server redirects (MOVED)	Standard protocol; redirect latency

 production default: smart clients (Redis Cluster drivers)
 with cluster-aware connection pools + topology refresh.

Replication Within the Cache Tier

 each shard gets replicas (Redis Cluster: master + replica/s):

 shard-a master ──async──► shard-a replica
 master dies → replica promoted (sentinel/cluster gossip)

 what replication buys HERE:
 - survive node loss WITHOUT cold-start stampede
 - maintenance failovers (upgrade replicas first)

 what it does NOT buy:
 - read scaling usually unnecessary (cache reads are cheap);
   consistency across copies is eventual — fine for caches,
   wrong assumption for locks!

Sizing a Fleet

 memory:  hot_set_GB × growth × safety(1.5) ÷ per_node usable
 shards:  throughput ÷ sustainable_per_node_rps (~50-100k simple ops)
          keep ≥ 25-30% memory headroom per node (fork/fragrance)
 example: 120GB hot set, 300k rps target
          → 6 × 32GB nodes ≈ comfortable, N+1 redundancy

Interview Framing

“Cache won’t fit on one machine” expects: sharded fleet via consistent hashing, virtual-node smoothing, resize impact quantified (“only 1/N remaps”), replication for survival-not-scaling, and sizing arithmetic. Distinguishing WHY replicas exist here (availability, not read load) is the detail interviewers listen for.