Golden Signals

The Four

 GOOGLE SRE's distilled dashboard canon:

 1. LATENCY:     how long do requests take?
                 p50 (typical) AND p99 (suffering tail) —
                 averages hide the users who hurt.
 2. TRAFFIC:     how much demand? req/s, msgs/s, concurrent users.
                 context for everything else (errors per second
                 mean different things at 10 vs 10k rps).
 3. ERRORS:      rate of failed requests — explicit (5xx) AND
                 implicit (200-with-wrong-content counts too!)
 4. SATURATION:  how "full" is the system? queue depths,
                 pool utilization, CPU/memory pressure.

 every service dashboard starts with these four;
 everything else is drill-down detail.

The Signals in Combination

 patterns that diagnose faster than any single signal:

 latency↑ + traffic flat        → dependency slowed / GC / lock
 latency↑ + traffic↑            → capacity ceiling approaching
 errors↑ + traffic flat         → bad deploy / dependency break
 errors↑ + saturation↑          → overload; shedding time
 ALL green + business metric ↓   → silent wrongness (hardest!)

 saturation deserves special respect:
 it's the LEADING indicator. queues fill before latency climbs;
 pools exhaust before errors spike. alert on saturation trends
 and you page BEFORE users notice instead of after.

Dashboard Design Per Signal

Signal	Panel essentials
latency	p50+p95+p99 lines, per-endpoint-class split
traffic	req/s + concurrent; annotate deploys!
errors	rate by code-class + business-error counters
saturation	pool/queue utilization with CAPACITY lines drawn

 deploy annotations on every panel:
 correlation between "when did it break" and "what shipped"
 answers itself when releases are visible on the timeline.
 missing this is free pain during every incident review.

From Signals to Alerts

 golden signals feed the alert hierarchy:

 □ SLO burn-rate alerts on error-signal (multiwindow — SRE lessons)
 □ saturation WARNINGS at 70-80% sustained (leading)
 □ latency alerts on p99-vs-SLO, not averages
 □ traffic ANOMALY alerts both ways (drops are incidents too!
   zero-traffic = dead pipeline, not good news)

 anti-patterns:
 ✗ one giant wall-of-graphs nobody can read at 3am
 ✗ alerting every signal equally → pager fatigue → real pages ignored
 ✗ per-instance dashboards as primary view (aggregate first,
   per-instance only when drilling into outliers)

Interview Framing

“Design the monitoring dashboard for this service” scored shape: four signals defined crisply with the implicit-errors subtlety named, combination-pattern diagnostics shown (at least two), saturation-as-leading-indicator point made, deploy annotations mentioned. Golden-signal questions test whether you monitor like an SRE or like a metrics collector — combinations over inventories.