Latency vs Throughput

Two Different Victories

• Latency: time for one unit of work — the user’s experience.
• Throughput: units of work per second — the system’s capacity.

They correlate until they don’t — and the divergence points are where design decisions live:

 ONE REQUEST'S JOURNEY              SYSTEM UNDER LOAD

 latency = wait + service           throughput ≈ service capacity
          ↑ queueing grows                    ÷ (1 - utilization)
            with congestion                   queueing tax explodes
            non-linearly                      near saturation

 light load: latency flat, throughput scales linearly
 near saturation: throughput plateaus while latency explodes
 → they are THE SAME PHENOMENON viewed from two sides

The Classic Trade: Batching

The clearest divergence example — database writes:

 INDIVIDUAL WRITES                 BATCHED WRITES (every 10ms)
 
 1000 writes/sec → 1000            buffer 10ms → flush together
 round trips + commits             one trip, group commit
 
 per-write latency: ~5ms           per-write latency: 5–15ms ✗ (worse avg)
 system overhead: high             system overhead: low ✓
 total throughput: lower           total throughput: much higher ✓

 paid: tail latency (a writer arriving just after a flush waits full window)
 earned: capacity to absorb 10x the write rate on same hardware

Kafka producers, group commit, connection pooling, vectorized execution — the entire industry runs on this trade.

Other Places the Tension Appears

Mechanism	Latency effect	Throughput effect
Batching	worse average, better under load	much higher
Caching	much better on hits	higher (backend relieved)
More replicas	better (parallel service)	higher
Stronger consistency (sync replication)	worse every write	lower
Connection pooling	slight queuing possible	much higher

Notice caching and replicas improve both — those are the moves to exhaust before accepting a genuine sacrifice.

Which One Wins When

 optimize LATENCY when the user is waiting interactively:
   checkout, ride match, search-as-you-type, first-screen paint
   → reject batching on the critical path; accept lower ceiling

 optimize THROUGHPUT when nobody is waiting synchronously:
   analytics, transcoding, backups, ETL, log ingestion
   → batch aggressively; utilization targets relax to 85%+

 the split IS the sync/async architecture line from data-flow:
 interactive paths stay unbatched; async pipelines batch everything

Measuring Both Honestly

Latency without percentiles lies (p50 can improve while p99 burns). Throughput without latency bounds lies harder (“50k rps!” at what p99?). Every capacity claim needs the pair:

 honest spec: "8k rps sustained at p99 < 200ms"
 not:         "fast" / "handles a lot"

Interview Framing

This tension surfaces whenever queues or batches appear. The scoring pattern: name which side the current component serves (“match path optimizes latency — no batching; ping ingestion optimizes throughput — batches everything”), and when proposing a batch anywhere, state its tail-latency price. Interviewers probe exactly there: “that 10ms flush window — who notices?”