Partitions

What Partitions Are

 a topic SPLIT into ordered, independent logs:

 topic "orders"
 ├── partition-0: e0 e3 e5 e8...   (ordered internally!)
 ├── partition-1: e1 e4 e6...
 └── partition-2: e2 e7...

 producer routes each event → ONE partition by KEY hash:
   partition = hash(user_id) % N

 guarantees live at partition scope:
 ✓ order preserved within one partition
 ✗ NO order across partitions (e2 may process before e0)

 everything about kafka-style systems follows from this split.

Why Partitioning Is the Scaling Story

 parallelism ceiling = PARTITION COUNT:

 [p-0]──► consumer-1 ┐
 [p-1]──► consumer-2 ├─ 3 partitions = max 3 busy consumers
 [p-2]──► consumer-3 ┘

 100 consumers on 3 partitions? 97 idle. 
 lag growing? first question: HOW MANY PARTITIONS?

 planning math:
   target throughput / per-consumer rate = minimum partitions
   peak 50k msg/s, consumer handles 500/s → ≥100 partitions

 headroom rule: over-partition generously (200 vs needed 100)
 because INCREASING later is possible but disruptive;
 keys must remap; ordering windows shift.

Key Selection: The Consequential Choice

Key	Property	Risk
user_id	user-scoped ordering ✓	celebrity users skew
order_id	near-perfect spread, no cross-entity order	hot? rarely
region	few keys → few hot partitions	terrible spread
random/null	round-robin perfect balance	zero ordering

 the tradeoff in one line:
 MORE MEANINGFUL THE KEY = BETTER LOCALITY/ORDERING,
 WORSE BALANCE RISK.

 mitigation for meaningful-but-skewed keys: 
 key-salting (append bucket suffix): user912#3 of #16 —
 spreads load, sacrifices full per-user ordering for it.

Partition Count Mechanics

 costs that scale WITH partition count:
 - open files/handles per broker (thousands × replication)
 - leader-election time during failures grows
 - end-to-end latency floor rises slightly
 
 too FEW partitions:
 - parallelism capped, lag unfixable by adding consumers
 - future resharding pain (see above)

 too many: modest overhead, mostly fine at thousands-scale.
 practical guidance: plan for 2–3× current need;
 thousands are normal for large platforms.

Interview Framing

“Design the Kafka topology for clickstream at 80k events/s” scored shape: partition-count derivation from throughput math (with consumer-rate assumption), key-choice tradeoff articulated (user_id for session ordering + salting escape hatch), the partitions-equal-parallelism-ceiling statement, and growth headroom reasoning. Topology questions are arithmetic wearing a diagram’s clothes — show the numbers.