Event Replay

What Replay Is

 the log RETAINS events after consumption (unlike queues).
 consumers hold POSITION (offsets), not ownership:

 [log] e1 e2 e3 e4 e5 e6 e7 ... e10000
        ▲                          
   rewind here → reprocess everything again

 REPLAY = resetting position and consuming history anew.

 impossible with queue semantics (message gone post-ack);
 routine with logs within their retention window.

The Killer Use Cases

 1. BUG-FIX REPROCESSING:
    analytics job had timezone bug for 2 weeks →
    fix code → replay those 14 days of events → corrected data.
    no data was lost; the CODE was wrong, not the events. 

 2. STATE REBUILDING:
    read-model corrupted/migrated → rebuild from source events:
    new consumer group from offset 0 → fresh projection ✓

 3. NEW CONSUMER ONBOARDED:
    fraud team wants 90 days of historical patterns →
    subscribe + consume retained history. zero producer changes.

 4. BACKFILL / MIGRATIONS:
    schema change downstream → replay through transformer
    writing new-format records.

Retention: Replay’s Fuel Gauge

 replay horizon = RETENTION window:

 - kafka default 7 days (time-based); size-based also common
 - infinite/compact topics for special streams (own lesson)
 - tiered storage extends cheaply to months

 design rule: retention ≥ your worst realistic
 "we need to reprocess" window.
 payment-event bugs discovered at day 9 with 7-day retention
 = permanent loss of replayability = manual repair hell.

 cost balance: retention is disk; tiered storage made
 long retention cheap. err LONG for business-critical streams.

Doing Replay Safely

 replaying 10M events into PRODUCTION downstreams is an event
 in itself — treat it like a deployment:

 □ THROTTLE: replay at fraction of live rate; watch downstream
   (DB writes, API calls) like a hawk
 □ ISOLATE when possible: separate consumer group → shadow
   output → validate → swap/copy
 □ IDEMPOTENCY assumed mandatory: replay WILL redeliver
   processed events by definition
 □ ORDERING windows: replayed traffic interleaves with live;
   per-key sequence guards prevent stale-overwrites-fresh
 □ TIME-BOX + MONITOR: lag dashboard open; abort switch ready

 replay storms have taken down downstream databases more
 often than any messaging outage. respect the throttle.

Replay vs Event Sourcing

 related but distinct — don't conflate in interviews:

 EVENT REPLAY:    operational technique on message logs;
                  reprocess SOME stream for SOME purpose
 EVENT SOURCING:  architecture where events ARE the database
                  of record; all state derived via replay
                  
 every event-sourced system replays constantly;
 most kafka systems replay occasionally.
 one lesson covers the technique here; ES gets its own.

Interview Framing

“A bug corrupted your recommendation model’s training data for a week” scored answer: identify replay as the recovery primitive (fix → replay window → rebuilt dataset), check retention covers the window FIRST (the gotcha!), then safety mechanics (throttle/idempotency/shadow-validate). Framing replay as “why logs beat queues for anything precious” ties the whole topic-model comparison together.