Message Brokers

What a Broker Must Do

 regardless of product, the job description:

 □ PERSIST messages durably (disk, replication)
 □ DELIVER to consumers with defined guarantees  
 □ BUFFER between mismatched rates (producer vs consumer)
 □ TRACK what's consumed (ack/offset bookkeeping)
 □ REDISTRIBUTE on failure (redelivery, consumer death)

 products differ in HOW — and those differences decide fit.

The Two Fundamental Models

 QUEUE MODEL (RabbitMQ, SQS):
   message → consumed ONCE → gone
   work-distribution semantics; competing workers

 LOG MODEL (Kafka, Pulsar, Kinesis):
   append-only log with OFFSETS;
   consumption = reading position; data RETAINED after read
   
   [log] e1 e2 e3 e4 e5 e6...
              ▲         ▲
           consumer A  consumer B   ← independent positions!
   
   multiple consumers read the SAME events at their own pace.
   replay = rewind offset. retention is a feature.

The Big Three Compared

Dimension	RabbitMQ	Kafka	SQS/Cloud queues
Model	smart broker/dumb queues	dumb broker/smart clients	managed queue
Routing	rich exchanges/bindings	topics+keys only	basic attributes
Replay	no	yes (retention window)	limited (DLQ redrive)
Throughput	~50k/s class	millions/s class	scales managed
Ordering	per-queue	per-partition strict	FIFO mode option
Ops burden	you run it	you run it (heavier)	none (managed)

Choosing By Workload

 RABBITMQ shines:
 - complex routing rules (topic exchanges, header matching)
 - per-message priorities, TTLs, dead-lettering richness
 - moderate throughput, low-latency task queues

 KAFKA shines:
 - event streaming at scale (analytics pipelines, CDC!)
 - multiple independent consumers of same events
 - REPLAY requirements (reprocess history after bug fix)
 - ordering per key (user's events in sequence)

 SQS-class shines:
 - standard work queues without ops burden
 - spiky/unpredictable volumes (managed elasticity)
 - teams without broker-operations capacity

Broker as Critical Infrastructure

 once adopted, the broker joins the availability path:

 - it going down stops ASYNC work (not usually user-facing reads)
   → still an incident: emails pile up, pipelines stall
 - HA topology required: clustered brokers, replicated logs,
   quorum writes for durability
 - capacity planning applies: disk for retention × rate,
   network for fan-out, partitions for parallelism

 monitoring essentials: depth trends, consumer lag,
 broker disk/CPU, end-to-end delivery latency.

Interview Framing

Broker-choice questions test model understanding over brand loyalty: distinguish queue-vs-log semantics FIRST (“consumers need the same events independently → log; work handed to one worker → queue”), then match to workload examples, then note managed-vs-self-hosted as a real team constraint. Saying “Kafka because scale” unexamined is the answer that gets dismantled gently.