Producer-Consumer

The Pattern

 [producer] ──push──► [ QUEUE ] ──pull──► [consumer(s)]
   makes work        holds it       processes it

 each message consumed by EXACTLY ONE consumer (vs pub-sub's
 broadcast — that's a different pattern, own lesson).

 producer and consumer:
 - don't know each other's identity
 - don't need to be up simultaneously  
 - scale independently

The Canonical Flow With Acknowledgments

 1. produce:   broker persists message, acks to producer
 2. deliver:   broker hands message to a consumer
 3. PROCESS:   consumer does the actual work
 4. ACK:       consumer tells broker "done"
 5. delete:    broker removes message

 THE CRITICAL DETAIL — ack comes AFTER processing:
   crash before ack → broker redelivers to another consumer
   → at-least-once semantics → consumers must be IDEMPOTENT

 ack BEFORE processing (the rookie bug):
   crash after ack, before work → message LOST forever.
   visibility-timeout systems make this mistake survivable,
   but why start wrong?

Competing Consumers: The Scaling Model

 one queue, N workers pulling in parallel:

 [queue] ══► worker-1 ┐
         ══► worker-2 ├─ throughput scales with worker count
         ══► worker-N ┘

 broker guarantees each message → ONE worker only
 (atomic delivery / lock during visibility window).

 this IS horizontal scaling for async work:
 lag growing? add workers. quiet night? shrink them.
 autoscaling signal = QUEUE DEPTH (textbook case).

Work Queue Design Choices

Choice	Options	Guidance
Delivery guarantee	at-most-once / at-least-once	At-least-once + idempotency
Ordering	none / per-key	FIFO only when business needs it
Visibility timeout	ms–minutes	> worst-case processing time
Poison handling	DLQ after N attempts	Always configure; own lesson

 visibility timeout subtlety:
 worker takes 30s; timeout is 10s → ANOTHER worker gets
 the same message mid-processing → duplicates galore.
 set timeout > p99 processing, plus heartbeat extensions
 for long tasks.

Where Producer-Consumer Fits

 ✓ image/video transcoding pipelines
 ✓ email/push notification dispatch
 ✓ order fulfillment steps
 ✓ report generation
 ✓ any WORK DISTRIBUTION problem

 not for:
 ✗ broadcasting events to many interested parties (pub-sub)
 ✗ event history/replay needs (logs/streaming)

Interview Framing

“Process uploaded videos asynchronously” scored skeleton: queue between upload-API and worker pool, competing-consumer scaling on depth metric, ack-after-process with idempotency note, visibility-timeout arithmetic vs transcode duration, DLQ for corrupt files. Complete flows with acknowledgment mechanics separate practitioners from diagram-sketchers.