Alert Design

The Prime Directive

 every page costs: sleep, trust, attention.
 alert fatigue = real pages ignored = incidents missed.

 THE TEST every alert must pass:
   would a human act DIFFERENTLY right now because of this?
   yes → page (or ticket if action can wait)
   no  → delete the alert

 alerts are for HUMANS; dashboards are for looking;
 logs are for searching. never blur these roles.

Symptom-Based Over Cause-Based

 SYMPTOMS (what users feel) → PAGE:
   error-budget burn rate (multiwindow)
   p99 latency vs SLO sustained breach
   checkout success-rate drop
   queue age exceeding freshness SLO

 CAUSES (what might explain symptoms) → TICKETS/DASHBOARDS:
   disk 80% full (warn; act in business hours)
   certificate expiring in 14 days (ticket)
   connection-pool utilization trending up (dashboard)

 cause-alerts as pages = paging on speculation:
   disk fills at 2am? maybe. meanwhile three such pages train
   the on-call that pages don't matter. reserve the pager.

Multiwindow Burn-Rate Alerts

 the SLO-native pattern for user-facing health:

 fast burn:  14.4× budget rate over 1h  (+5m confirm)
             → page: "SLO gone within ~2 days at this pace"
 slow burn:  6× over 6h                  (+30m confirm)
             → page next-day priority

 properties that make it work:
 □ CONFIRMATION WINDOWS kill flapping single-spikes
 □ severity matches URGENCY not ego
 □ every page links runbook: what to check, what to try,
   when to escalate — unwritten runbook = unfinished alert

 anti-patterns to hunt quarterly:
 ✗ threshold-on-raw-CPU pages
 ✗ per-instance alerts (aggregate first!)
 ✗ anything firing >2×/month without action taken

Alert Lifecycle Discipline

Stage	Practice
creation	requires: query, threshold, rationale, runbook link
review	monthly noisy-alert report; top offender gets fixed or killed
after incidents	new alert only with postmortem-backed hypothesis
deletion	alerts retire like code; dead ones accumulate noise

 the meta-metric worth tracking: PAGE PRECISION —
 fraction of pages leading to genuine action. below ~50%
 means the system cries wolf and humans have stopped listening.
 manage it like an SLO of its own.

Interview Framing

“Design alerting so on-call doesn’t burn out” scored shape: prime-directive test stated, symptom-vs-cause split with concrete examples of each destination, multiwindow burn-rate mechanics shown, lifecycle/noise-management practices included, precision-as-SLO meta-metric named. Alert-design questions test empathy under load — the best answers optimize for the 3am reader, not the dashboard author.