Headroom & Utilization

The Counterintuitive Target

Full utilization looks efficient and operates terribly:

 latency vs utilization (queueing behavior)

 p99 latency ▲                    ██ 20x+
             │               █████
             │          █████        ← queues grow super-linearly
             │      ████                as service nears saturation
             │ ████
             └──────────────────────────► utilization
               50%   60%   70%   80%   90%  95%+

 at ~50% utilized, latency ≈ 2x raw service time — calm
 at ~95%, latency ≈ 20x — every request queues behind everyone

The “wasted” capacity between 70% and 100% is not waste — it is the shock absorber that keeps tail latency alive.

What Headroom Must Absorb

Provisioned capacity simultaneously covers:

 1. PEAK traffic        (2–3x average on diurnal curves)
 2. NODE LOSS           (N+1: fleet minus one still serves peak)
 3. DEPLOYS             (rolling restarts temporarily remove capacity)
 4. RETRIES/REPLAY      (failures amplify demand exactly during incidents)
 5. GROWTH              (weeks of organic increase before procurement lands)

 sizing to #1 alone means #2 or #3 becomes an outage multiplier

The N+1 Rule

Minimum redundancy arithmetic:

 requirement: serve peak load P with any single node dead
 nodes needed = ceil(P / node_capacity) + 1

 example: P = 4k rps, node = 300 rps sustained target
          ceil(13.3) + 1 = 15 → deploy 16 (even count helps rolling waves)

 stateful systems add a second axis: replica factor ≥ 3 so one loss
 keeps quorum AND capacity

Utilization Targets by Tier

Tier	Sustained max	Rationale
Payments/critical path	50–60%	Must absorb peak + loss + deploy together
Core APIs	~70%	Standard slack; alert threshold here
Cache tier	~70% memory	Evictions spike past that
Batch/analytics	85–90%	Latency-tolerant work can queue

Alert placement follows targets: warn at sustained target, page when trending past it — because by 90%+, the incident has already begun.

Autoscaling Changes Timing, Not Targets

Elastic tiers can run tighter averages, but reaction lag dictates floors:

 scale-out lag: minutes (instance boot + warmup + health)
 spike rise time: seconds
 
 → keep a floor provisioned ≈ expected minimum load × small buffer;
   let autoscaler ride the diurnal curve above it
 → databases/caches (stateful, slow to scale) keep full headroom policy

Interview Framing

Utilization discipline signals production experience. Score pattern: after any node-count division, state the utilization assumption (“dividing by per-node sustained capacity at ~70%, not nameplate”), apply N+1 once, and note which tier gets stricter targets. When interviewers ask “why not fewer nodes?” — the answer is this lesson: the missing machines are the reliability budget.