The Latency Hierarchy

Why One Table Rules System Design

Every architectural decision — cache placement, replication mode, region strategy — is secretly a decision about this table. Latencies differ by orders of magnitude across the stack, and designs that ignore the gaps build castles on physics they never checked.

 APPROXIMATE LATENCIES (order of magnitude; modern commodity hardware)

 OPERATION                          TIME          SCALED TO 1s = 1 CPU CYCLE
 ───────────────────────────────────────────────────────────────────────────
 L1 cache reference                 ~1 ns         1 second
 Mutex lock/unlock                  ~25 ns        25 seconds
 Main memory reference              ~100 ns       2 minutes
 Compress 1KB (snappy-class)        ~3–10 µs      1–3 hours
 Send 2KB over 1 Gbps network       ~20 µs        6 hours
 SSD random read                    ~150 µs       2 days
 Read 1MB sequentially from SSD     ~1 ms         12 days
 Disk seek (spinning HDD)           ~10 ms        4 months
 Cross-AZ round trip                ~1–2 ms       2–4 weeks
 Cross-region RTT (US coast-coast)  ~50–70 ms     ~2 years
 Cross-continent RTT (US↔EU)       ~100–150 ms   ~5 years

(Scale column after Dean’s classic “Latency Numbers Every Programmer Should Know”; absolute values drift with hardware, ratios endure.)

The Gaps That Design Decisions Live In

Gap	Ratio	Decision it dictates
Memory vs SSD	~1000x	Hot working sets belong in RAM (caches exist because of this)
SSD vs spinning disk	~100x	Random-access workloads never go on HDD
Same-AZ vs cross-AZ	~10x within DC	Replica placement changes write latency measurably
Cross-region vs same-region	~50–100x	Sync replication across regions is usually unacceptable
Function call vs network call	~10,000,000x	In-process composition beats chatty microservice calls

The last row explains a thousand microservices regret stories: what was a nanosecond function call became a millisecond network hop — a million-fold tax per call.

Reading the Table as Architecture

 "p99 under 50ms globally"
 → cross-continent RTT alone is 100ms+
 → impossible from one origin → CDN/regional deployment REQUIRED,
   not optional

 "strong consistency for payments across US + EU"
 → sync quorum across 100ms+ RTT
 → every payment write pays ≥ one ocean crossing
 → acceptable ONLY because payments tolerate the latency bill

 "cache hit ratio matters more than anything"
 → memory ~100ns vs SSD ~150µs: hits are ~1000x cheaper than misses
 → a cache at 90% hit is doing nearly all the work

Sequential vs Random Access

One subtlety worth memorizing: sequential reads are dramatically cheaper than random ones on both disks and networks.

• Spinning disk: seek 10ms, then read sequentially fast → random access pattern kills throughput.
• Networks: per-message overhead dominates small payloads; batching amortizes it.

This single fact powers batching everywhere — log-structured storage, group commit, vectorized scans.

Interview Framing

Interviewers test this table indirectly: “why not store sessions in Postgres?” (memory-vs-disk gap), “can we do sync writes to EU?” (ocean math). Candidates who answer with orders of magnitude — “that’s a 1000x gap, so no” — demonstrate fluency that no framework name can fake. Internalize five anchor points: L1 ~ns, memory ~100ns, SSD ~100µs, cross-AZ ~1ms, cross-ocean ~100ms.