p50, p95, p99 — Percentile Latency

The Language of Latency

Percentiles describe a distribution by its cutoffs:

 p50 (median): half of requests are faster than this
 p90:          9 in 10 requests are faster
 p95:          19 in 20 faster — the "typical worst" most users hit
 p99:          99 of 100 faster — the tail that power users live in
 p99.9:        the tail that pages on-call

 EXAMPLE DISTRIBUTION (1,000 requests):
 p50 = 40ms   typical experience, marketing-friendly
 p95 = 120ms  slow-ish but acceptable
 p99 = 800ms  one user in a hundred suffers THIS
 p999 = 3.2s  someone is having a terrible time right now

Why the Average Is the Dangerous Statistic

 100 requests: 95 complete at 50ms, 5 hang at 2 seconds

 average = (95×50 + 5×2000)/100 ≈ 147ms   ← looks tolerable
 reality  = 95% of users see 50ms; 5% see 2 SECONDS

 the average describes NOBODY who actually used the system
 worse: averages hide bimodality entirely — cache-hit/fast vs
        miss/slow populations average into one meaningless number

SLOs written on averages systematically underprotect the users having problems — which are the only users whose opinion of you forms that day.

Tail Math: Aggregation Makes Tails Worse

A page calling many backends inherits everyone’s tails:

 one backend call at p99 = 500ms → 1% of requests slow

 page makes 10 parallel calls:
 P(all 10 fast) = 0.99^10 ≈ 0.904
 → ~10% of page loads hit SOME call's tail
 
 30 calls: 0.99^30 ≈ 0.74 → a quarter of loads degraded
 
 microservice fan-out turns rare tails into common experiences —
 this is why tail latency owns distributed-systems attention

Setting Targets Honestly

Target	Meaning	Typical use
p50 under X	Median feel	UX baselines
p95 under X	Most users’ worst case	Standard API SLO
p99 under X	Tail discipline	Revenue paths
p99.9 under X	Near-total coverage	Payments, matching

Two rules make targets real:

1. Measure per endpoint, not system-wide blends — blending hides the slow endpoint among fast ones.
2. Measure from the user’s vantage when possible (client-side telemetry); server percentiles exclude network legs users experience.

The Cost of Chasing Nines of Latency

Like availability, each percentile improvement costs multiples:

• p50 → p95: usually caching + indexing.
• p95 → p99: eliminating GC pauses, connection pool tuning, removing remaining disk reads.
• p99 → p99.9: hardware isolation, avoiding noisy neighbors, hedged requests.

Hedged requests (send duplicate after slight delay, take first answer) are the classic tail-slaying trick — they trade throughput for tail.

Interview Framing

Any latency number stated without a percentile is an invitation for pushback (“do you mean median or p99?”). Strong candidates volunteer both: “targeting p95 under 300ms; p99 we accept around 600ms because history is not the money path.” When fan-out appears in a design, computing the 0.99^N math aloud is a reliable seniority marker.