Per-Key Ordering

The Compromise That Wins

 global ordering: correct but serialized (useless throughput)
 no ordering:     fast but breaks stateful sequences

 PER-KEY: order guaranteed WITHIN each key's stream;
 keys flow independently in parallel:

 key=A: e1 ─► e2 ─► e3          (strict ✓)
 key=B: e1 ─► e2                (strict ✓)
 key=C: e3 ─► e1 ─► e2          (strict ✓)
 A vs B vs C: any interleaving. don't care. ✓

 10k keys → thousands of independent ordered streams →
 full partition parallelism WITH per-entity correctness.

Implementation Mechanics

 producer side:
   partition = hash(key) % partitions
   same key ALWAYS lands same partition → its order preserved

   [p0: users {1,4,7}] [p1: users {2,5,8}] [p2: users {3,6,9}]
        user-1's events strictly ordered within p0 ✓

 requirements for the guarantee to actually hold:
 1. SINGLE WRITER THREAD per key (or external sequencing)
    two threads emitting for user-912 race at produce time!
 2. stable hash mapping (partition count fixed; changes remap)
 3. consumer processes partition serially
 4. retry-redelivery reordering handled (sequence numbers)

Choosing Keys: Domain-Driven

 ask: "which entities have ORDERED STATE MACHINES?"

 domain              | right key         | why
 --------------------|-------------------|------------------
 bank account        | account_id        | balance sequence
 chat conversation   | conversation_id   | message sequence
 user profile update | user_id           | last-write ordering
 inventory SKU       | sku_id            | stock decrements
 IoT device          | device_id         | reading sequence

 WRONG keys look reasonable but break semantics:
 - order_id for payment events: payments span orders? no wait—
   one order's payments ordered... unless refunds cross keys.
 - region/country: too few distinct values = hot partitions.

 rule: the ENTITY whose state evolves = the key.

When Per-Key Isn’t Enough

 ordering needs that CROSS keys:

 - "user's orders" ordered but keyed by order_id ✗
   → denormalize: embed user_id in events, dual-write both keys,
     or accept per-order ordering only
 - workflow spanning services/entities
   → orchestrator serializes via its own state store instead
 - truly total order required
   → single sequencer/leader per stream (rare, costly)

 detect these EARLY: they reshape data models, not just configs.

Skew: The Per-Key Tax Revisited

 hot keys concentrate into one partition:
 celebrity user → their partition melts, others idle.

 escape hatches:
 □ salting: key#bucket (splits ordering — partial loss)
 □ TWO-LEVEL: hot entity gets dedicated partition + path
 □ isolate: route known-hot producers to separate topic
 
 decide CONSCIOUSLY: which matters more,
 this entity's strict order OR cluster balance?
 sometimes answer differs per event type!

Interview Framing

“Chat messages must arrive in order at scale” scored answer: per-key with conversation_id as THE design (never global), the four-holdings-it checklist (single writer, stable hash, serial consume, seq-guarded retries), skew acknowledgment with salting tradeoff named, and the cross-key follow-up pre-empted (per-conversation suffices; no cross-chat ordering exists). This lesson IS the interview answer most ordering questions want.