Replication Lag

What Lag Actually Is

 LAG = time between primary commit and replica apply.

 timeline of one transaction:
 T0: commit on primary ──ship──► T1: arrives at replica
                                ──apply──► T2: visible
 
 user-visible lag ≈ T2 − T0   (normally ms; incidents = seconds+)

 during the window:
 primary says X, replicas say old-X.
 EVERY read routed to a replica sees the past.

The Cause Catalog

 □ write burst > replica apply rate     (bulk import day)
 □ ONE LONG TRANSACTION blocks everything behind it
    (replicas replay in commit order!)
 □ replica under-provisioned vs primary (cheap-replica trap)
 □ network saturation (cross-region replication)
 □ single-threaded applier on wide write mixes
 □ table rewrites/migrations (ALTER on 500M rows)

 the long-transaction row surprises everyone once:
 a Friday migration froze all replicas for 40 minutes.
 batch your migrations. always.

Measuring It Properly

 per-replica gauges + alerting:

 - postgres: pg_last_wal_replay_lsn delta → bytes/seconds behind
 - mysql:    Seconds_Behind_Source (coarse but standard)
 - managed:  native metrics (CloudWatch/Azure Monitor equivalents)

 ALERT thresholds that work:
   warn at p95 lag > 2s      (capacity review)
   page  at lag > 30s        (user-visible staleness incoming)
 
 ALSO measure application-side: write→read-back probes
 detect routing bugs pure server metrics miss.

Mitigation Stack

 PREVENTION:
 - provision replicas ≥ primary specs (they also serve reads!)
 - break long transactions into batches (<1s each)
 - parallel-apply enabled where supported
 - dedicated replication NIC/bandwidth cross-region

 COPING:
 - lag-aware read routing: skip lagging replicas for
   freshness-sensitive paths
 - read-your-writes windows (sticky-to-primary N sec post-write)
 - LSN-token waits for correctness-critical reads
 - degrade gracefully: show "updated moments ago" UI affordances

 none of these remove lag — they make it INVISIBLE
 to the users who'd notice.

When Lag Is a Feature Requirement

 design reviews should ASK per feature:
 "what's your staleness budget?"

 product pages:        minutes OK    → async replicas fine
 order history:        seconds       → sticky-window needed  
 inventory at checkout: ZERO         → primary reads only
 analytics:            hours OK      → warehouse, don't care

 making teams STATE the number surfaces mismatches early —
 the alternative is discovering them via support tickets.

Interview Framing

Lag questions test operational depth: cause catalog (long-transaction story lands well), monitoring with concrete thresholds, the three-tier mitigation stack (prevent/copy/route), and staleness-budget-per-feature framing. Candidates who treat lag as purely an ops problem miss that it’s a DESIGN input — say that sentence.