Log Compaction

Time Retention vs Key Retention

 default retention: TIME/SIZE based
   "keep 7 days" → old events deleted regardless of content

 LOG COMPACTION: KEY-based retention
   per key, keep AT LEAST the LATEST value; clean history:

 before compaction:
   [u1:v1] [u2:v1] [u1:v2] [u3:v1] [u2:v2] [u1:v3]

 after compaction:
   [u2:v2] [u3:v1] [u1:v3]
   
 every key survives with its newest state;
 superseded versions garbage-collected.

The State-Broadcast Use Case

 compacted topic = REPLAYABLE SNAPSHOT STREAM:

 use case: service needs CURRENT config/state of all entities:

 new consumer starts → reads WHOLE compacted topic from 0
 → ends holding latest value of EVERY key = full current state!
 
 [compacted "user-profiles" topic]
 u1:{name:A} u2:{name:B} ... (latest only)
        │
 new consumer replays in seconds-minutes → in-memory table ✓

 this is how kafka ecosystems bootstrap:
 change events stream live + compacted history = catch-up mechanism.
 kafka's own __consumer_offsets runs compacted. dogfooding.

Where It Shines

Stream	Compacted content
Entity change events	latest state per entity id
Config/feature flags	latest setting per flag
Schema/metadata	latest schema per subject
Id→location mappings	latest routing entry

 common thread: SEMANTICS OF A TABLE, mechanics of a log.
 compacted topics are tables wearing log costumes —
 the changelog pattern that underpins stream processing
 (kafka streams tables, materialized views over streams).

Operational Notes

 compaction is ASYNCHRONOUS background cleaning:
 - disk temporarily holds uncompacted data (tombstone windows!)
 - TOMBSTONES (null values) delete keys — kept briefly,
   then purged WITH their key (delete.propagation.ms)
 - never assume compaction INSTANT after write
 
 sizing: segment-based; cleaner throttles to protect IO.
 monitor: dirty-ratio, cleaner lag.

 gotcha: consumers must tolerate DUPLICATES + ordering quirks
 even in compacted topics (at-least-once still applies!).
 last-write-wins readers are the natural fit.

Interview Framing

“How would a newly deployed service learn current state of all million users without hammering your DB?” scored answer: compacted topic as replayable-state-stream (read-from-zero = snapshot), contrast with time retention explicitly, changelog/table framing, tombstone semantics mentioned. This question appears constantly because it separates people who’ve operated kafka’s model from those who’ve only produced/consumed blindly.