What Is Caching

The Definition

 CACHE: a fast store of COPIES, kept close to where they're used,
        holding answers to questions likely to be asked again.

 request flow:
   check cache ──HIT──► return copy          (~1 ms)
        │
       MISS ──► fetch from source ──► store copy ──► return
                (~100 ms)              (for next time)

The Economics

 value = hit_rate × (slow_path_time − fast_path_time) − staleness risk

 worked example:
   10k rps, 90% hit rate, db read 80ms vs redis 1ms
   → 9k rps never touch the db; p50 latency drops 79ms
   → db provisioned for ~1k rps instead of 10k

 caching is usually the CHEAPEST scalability lever available:
   no sharding, no replicas — just remembering.

Why It Works: Locality

 traffic is predictably skewed:
 - popularity follows power laws: top 20% of items ≈ 80%+ of reads
   (products, videos, profiles, routes)
 - temporal locality: what was just read is likely read again soon
   (a viral post's first hour)

 caching exploits BOTH: keep hot data hot, recent data ready.
 uniform random access = caching barely helps (rare in practice).

Hit Rate Is the Whole Game

 hit rate depends on:
 - CAPACITY      too small → thrashing evictions
 - TTL           too short → constant misses; too long → stale
 - KEY choice    fragmented keys (per-user variants) split heat
 
 measure per-endpoint:
   GET /trips/:id     hit rate 97%   ← great
   GET /search?q=...  hit rate 12%   ← maybe don't cache this
 
 low-hit-rate caches still cost memory + invalidation complexity
 for near-zero benefit. not every endpoint deserves one.

Where Caches Hide

 every layer of a modern stack caches:

 browser         images, JS bundles          (Cache-Control headers)
 CDN             static + semi-static pages  (edge locations)
 API gateway     auth decisions, responses   (short TTLs)
 application     computed objects, sessions  (Redis/Memcached)
 database        buffer pool, query plans    (automatic)
 CPU             instructions, memory lines  (hardware)

 each layer trades freshness for speed independently —
 understanding ALL of them is required to debug
 "why does the user see old data?"

The Price You Pay

Cost	Nature
Staleness	Copy diverges from source
Invalidation	Hard problem — knowing when copies die
Memory cost	RAM is real money at scale
Complexity	Two sources of truth on every read path
Cold starts	Empty cache after deploy/failover = origin stampede

The invalidation column has its own lessons ahead. The honest summary: caching converts compute problems into consistency problems.

Interview Framing

Every design gets “where’s the cache?” Scored shape: identify the top reads by volume, claim hit rates with rough justification (“profiles: 95% — power-law popularity”), size it (“1% of dataset hot set ≈ 8GB → single Redis”), and name the invalidation strategy. Caching mentioned WITHOUT hit-rate reasoning is noise; with it, engineering.