Consumer Groups

The Model

 CONSUMER GROUP = a named set of consumers sharing the work.
 each PARTITION → exactly ONE member of the group at a time.

 topic with 6 partitions, group "billing":
 
 [p0]──► c1    [p3]──► c2
 [p1]──► c1    [p4]──► c2
 [p2]──► c1    [p5]──► c2

 DIFFERENT groups read independently:
   group "billing"  at offsets (12, 15, 9...)
   group "analytics" at offsets (40, 41, 38...)
 same events, separate positions, no interference.
 = pub-sub AND work-queue semantics from one primitive!

Scaling Within a Group

 add members → partitions redistribute:

 3 partitions:
 1 consumer:  [p0 p1 p2]──► c1
 2 consumers: [p0 p1]─►c1  [p2]─►c2
 3 consumers: [p0]─►c1 [p1]─►c2 [p2]─►c3
 4 consumers: [p0]─►c1 [p1]─►c2 [p2]─►c3 [IDLE]─►c4 ⚠

 THE CEILING AGAIN: members beyond partition count idle.
 lag won't drop by adding consumer #7 to 6 partitions.

 autoscaling rule for kafka consumers:
   max_replicas = partition_count. hard ceiling. plan partitions.

Rebalancing: The Necessary Evil

 membership changes trigger REBALANCE:
 - consumer joins (deploy scale-up)
 - consumer leaves/dies (crash, network blip, slow heartbeat)
 - subscription changes

 during rebalance:
 ✗ ALL consumption in group typically PAUSES
 ✗ can take seconds to MINUTES (large groups)
 ✗ in-flight processing interrupted

 rebalance storms happen when:
 crash-looping consumers + long processing times +
 aggressive session timeouts feed each other:

 slow consumer → kicked → rebalance pauses all →
 lag grows → restart joins → another rebalance...

Taming Rebalances

 □ RIGHT-SIZE timeouts: session/heartbeat vs actual
   processing time (max-poll-interval must exceed
   your batch processing worst case!)
 □ STATIC MEMBERSHIP: stable instance IDs survive restarts
   without triggering rebalance (rolling deploys!)
 □ COOPERATIVE-STACKED protocol: incremental moves,
   unaffected partitions keep flowing (modern default)
 □ AVOID frequent scaling flaps: hysteresis on autoscaler

Assignment Strategies

Strategy	Behavior
Range	per-topic contiguous blocks; skew-prone multi-topic
RoundRobin	even spread across topics
Sticky	preserves assignments across rebalances ✓
Cooperative-sticky	sticky + incremental handoff ✓✓

 default to cooperative-sticky unless legacy constraints;
 range strategy's skew is a classic debugging rabbit hole.

Interview Framing

“Scale Kafka consumption for growing order volume” scored shape: group model explained via one-partition-one-member invariant, the partition-ceiling called out explicitly with idle-consumer diagram, rebalance mechanics volunteered as the operational trap (with static-membership and cooperative fixes), assignment strategy named. The ceiling awareness plus rebalance wariness signals real kafka production experience.