Message Ordering

The Assumption vs The Reality

 producer sends: A, B, C

 naive expectation: consumers process A, B, C — always.
 
 distributed reality breaks this everywhere:
 - parallel consumers finish out of order (C beats A)
 - retries deliver B twice, late
 - multiple producers interleave unpredictably

 ORDERING IS EXPENSIVE. every guarantee has a price:
 understand exactly which order you NEED before paying.

The Ordering Hierarchy

 GLOBAL total order:   everyone sees ALL messages in one sequence
                       → single serialization point; kills throughput
 
 PARTIAL/per-key:      order preserved WITHIN a key's stream only
                       (user-912's events ordered among themselves)
                       THE practical default (kafka partitions)

 NO order:             any arrival sequence acceptable
                       (independent work items)

 asking "do I need ordering?" usually means:
 "which EVENTS must not reorder relative to WHICH others?"
 — the answer is almost never "all of them."

Why Order Breaks Even When Promised

 per-partition ordering holds ONLY under these conditions:

 1. ONE producer thread per key (parallel writers race!)
 2. key maps to ONE partition (stable hashing)
 3. consumer processes partition SERIALLY
    (one worker per partition at a time)
 4. failures handled by redelivery REORDER nothing globally
    (retry B after C already ran = reordering!)

 condition 4 is the silent killer:
 at-least-once delivery + retries = local reordering
 UNLESS consumers detect/handle it (version numbers, timestamps).

Designing For Ordered Processing

 pattern: SEQUENCE NUMBERS + buffering:

 each key's events carry monotonic seq:
   {user:912, seq:41}, {user:912, seq:42}

 consumer tracks last-seen per key:
   expected 42, got 44 → BUFFER 44, wait for 42 (gap detected!)
   got duplicate 41     → ignore (already processed)

 bounded wait: if 42 never arrives (lost), timeout + skip-with-log.
 this converts "delivery order chaos" into "detectable, fixable gaps."

When You Actually Need It

Scenario	Needs ordering?
User’s chat messages	YES per conversation
Clicks for analytics	No
Account state changes	YES per account
Image transcoding jobs	No
CDC database replication	YES per row

 the anti-pattern: demanding global FIFO "to be safe."
 cost: serialization bottleneck, no consumer parallelism,
 fragile failure modes. safety comes from IDEMPOTENCY +
 version checks, not from artificial ordering.

Interview Framing

“Messages must arrive in order” scored response: interrogate the requirement (“ordered per WHAT?”), offer per-key partitioning as the scalable answer, then expose the retry-reordering trap with the sequence-number-buffering fix. Candidates who accept global-ordering requirements uncritically get asked to explain their throughput ceiling — have that conversation first instead.