Autoscaling

The Concept

 AUTOSCALING: add capacity as demand rises, remove it as it falls.

 load curve (diurnal):          provisioned capacity:
        ▄▄█▄▄                          ▄▄█▄▄   ← tracks demand
     ▄▄▀▀   ▀▀▄▄                    ▄▄▀▀    ▀▀▄▄
 ▀▀▀▀            ▀▀▀▀           ▀▀▀             ← never idle overpaying

 without autoscaling you provision for PEAK always:
 paying for sleeping nodes ~20h/day. with it: pay ≈ what you use.

Trigger Metrics

Metric	Good for	Caveat
CPU utilization	Compute-bound services	Misleading for IO-bound
Requests per instance	Traffic-proportional work	Needs RPS signal plumbing
Queue depth	Worker pools	Direct measure of backlog
p99 latency	User-experience truth	Noisy; reacts late
Custom (active trips, conns)	Domain accuracy	You own the tuning

 best practice: scale on the resource closest to the actual
 constraint. workers → queue depth; APIs → rps-per-pod or cpu;
 anything user-facing → watch latency as the backstop alarm.

Policy Mechanics

 target tracking (simplest, recommended):
   "keep cpu at 55%; add/remove pods to hold it"
 
 step scaling:
   "+1 pod per +500 rps above baseline"
 
 scheduled scaling:
   pre-warm before known events (sale at 9am, game launch)
   — the fix for predictable spikes reactive rules miss

 scale-out is fast; scale-in must be SLOW and careful
 (see traps below)

The Traps

 1. FLAPPING
    threshold crossed → add node → load drops below → remove node
    → loop forever. fix: COOLDOWNS (e.g., 3–5 min between actions)
    + hysteresis (scale-in threshold well below scale-out's)

 2. SCALE-IN KILLS IN-FLIGHT WORK
    removing a node mid-request/queue-drain = errors.
    fix: graceful drain — stop accepting, finish work, then die;
    long jobs need checkpoint/migration instead

 3. COLD START LATENCY
    new pod needs image pull + boot + cache warm: 30–90s blind spot.
    fix: overprovision slightly, scheduled pre-scaling,
    keep images small, warm caches on boot

 4. SCALING THE WRONG LAYER
    app scales fine but the DB behind it saturates →
    autoscaling multiplies pressure on a fixed bottleneck.
    find the real constraint first (bottleneck lesson)

What Autoscales Well vs Poorly

 ✓ stateless API tiers         the canonical use case
 ✓ queue worker pools          depth-driven, self-correcting
 ✗ primary databases           vertical + read replicas instead
 ✗ stateful connection servers rebalancing live sessions is hard
 ✗ anything needing stable identity/IP (licensing, allowlists)

Stateful tiers scale by partition-rebalancing machinery, not by naive pod counts — different mechanism entirely.

Interview Framing

Mention autoscaling with specifics to score: trigger metric chosen per tier (“workers on queue depth”), cooldowns named, cold-start acknowledged with pre-warming, and the honest boundary — “database does NOT autoscale; it gets replicas and sharding.” That last sentence separates operators from tutorial-followers.