Partitioning

The Definition

 PARTITIONING: divide a dataset into disjoint subsets,
 each owned by a different node.

 node-1: trips A–F        ┐
 node-2: trips G–M        ├─ together = the whole dataset
 node-3: trips N–Z        ┘

 vs replication (copies): partitions are COMPLEMENTARY halves
 of the same story. production topology: many partitions,
 each replicated to K nodes.

Why Single Nodes Stop Being Enough

 walls hit in rough order:
 - working set exceeds RAM → cache misses dominate latency
 - write IOPS saturate disk regardless of instance size
 - replication lag grows with write volume per primary
 - backup/restore windows become unacceptable (hours)
 - indexes bloat; queries degrade even with perfect tuning

 reads scale via replicas. WRITES and DATASET SIZE don't —
 only partitioning removes the single-node ceiling.

Partitioning Dimensions

 BY ENTITY (horizontal / sharding):
   users 1M–2M → node A, users 2M–3M → node B
   rows split across machines ← this lesson's focus
 
 BY FEATURE (functional):
   orders DB, inventory DB, sessions DB — different services own
   different datasets entirely (microservice-style separation)
 
 BY ACCESS PATTERN:
   hot recent data on fast storage, cold history archived
   (time-based partitioning within one logical dataset)

Interviews mean entity-partitioning (“sharding”) unless stated otherwise, but naming all three shows range.

The Routing Question

Every partitioned system needs: given key X, which node?

 approaches:
 RANGE:      user_id 1–1M → node1         (own lessons)
 HASH:       hash(user_id) % N → node     (own lessons)
 DIRECTORY:  lookup table key→node        (flexible, extra hop)
 CONSISTENT HASHING: ring assignment that minimizes movement
             during resharding            (own lesson)

 routing lives in: client library, proxy tier (Vitess/Citus),
 or coordinator nodes (Cassandra/DynamoDB style)

What Gets Hard

Problem	Nature
Cross-shard queries	”Top 10 by score” touches every shard
Cross-shard transactions	Two-phase commit territory; avoid by design
Hot partitions	Celebrity user melts one node
Rebalancing	Adding nodes moves data under traffic
Secondary access	Query by email but sharded by id → scatter-gather

Each has established mitigations — and each is its own lesson later. The meta-point: partitioning converts ONE hard problem into SEVERAL small ones plus coordination.

The Golden Rule

 choose the partition KEY so that the queries you run most
 touch exactly ONE partition:

 sharded by user_id → user's orders, cart, profile: single-shard ✓
                     global trending list: cross-shard ✗ (different tool)

 data modeling BEFORE infrastructure: the key choice IS the design

Interview Framing

“When does sharding enter the picture?” — after vertical and replicas fail against WRITE growth or dataset size, quantified. Then immediately: partition scheme + key rationale for THIS workload’s dominant queries. Candidates who shard without justifying the key get the cross-shard-query follow-up they can’t answer.