Offset Management

Offsets as Progress Bookmarks

 log model: consumption position = an OFFSET per partition.

 [e1 e2 e3 e4 e5 e6 e7 e8]
            ▲ committed offset (5)
 → restart consumer: RESUME from 6. nothing lost, nothing repeated.

 COMMITTING = telling the group coordinator "processed through N."
 WHERE/WHEN you commit = your actual delivery guarantee.

Commit Timing = Guarantee Selection

 AT-LEAST-ONCE:
   process batch → THEN commit
   crash pre-commit → redelivery of processed batch (dupes)
   
 AT-MOST-ONCE:
   commit FIRST → then process  
   crash post-commit pre-process → messages skipped (loss)
   
 EFFECTIVELY-EXACTLY-ONCE:
   offsets + output in ONE atomic transaction
   (kafka transactions; or dedupe-table equivalent)
   crash → all-or-nothing replay

 same three-way fork as delivery guarantees — because
 offset timing IS where that guarantee is implemented.

The Batch Size Tension

 commit every message:      safest, slowest (commit = RPC)
 commit every 1000:         fast, bigger replay window on crash
 
 window sizing logic:
 - crash frequency × window size = typical duplicate volume
 - idempotent consumers make windows CHEAP to widen
 - latency-sensitive pipelines: tighter windows
 
 production default: auto-commit disabled, explicit commits
 per batch with size/time bounds. know exactly what's committed.

The Wrong-Commit Recovery Playbook

 MIS-committed offsets happen (buggy deploy, manual error):

 SCENARIO: committed too FAR (skipped unprocessed events)
   → data gap: events never processed!
   RECOVERY: reset offsets BACKWARD to before gap,
             consumers re-read (idempotency makes this safe ✓)

 SCENARIO: committed too LITTLE (reprocessing done work)
   → duplicate wave: harmless IF consumers idempotent
     otherwise: incident.

 TOOLS: kafka-consumer-groups --reset-offsets 
        (--to-datetime / --shift-by / --to-earliest)
 practice BEFORE the 2am emergency requires it.

Monitoring Offsets

 □ lag per partition (offsets ARE the lag metric source)
 □ commit rate vs consume rate mismatch alerts
 □ last-commit-age per group (stalled groups visible!)
 □ offset RETENTION: idle group's offsets can EXPIRE
   (default windows!) → silent reset-to-latest on revival
   → massive data skip. pin retention for paused-but-precious groups.

Interview Framing

“How do you ensure no order events are lost during consumer deploys?” scored path: offset-commit semantics as THE mechanism, at-least-once chosen via process-then-commit, deploy-time behavior traced through it (old dies mid-batch → redelivery), idempotency closing the loop, plus the offset-expiry gotcha for good measure. Offset fluency is kafka literacy’s core test.