Metrics Instrumentation

The Metric Types

 four primitives cover nearly everything:

 COUNTER: only goes up (requests_total, errors_total)
   → rates via increase() over windows. THE workhorse.
 GAUGE:   point-in-time value (queue_depth, active_connections)
   → watch absolute levels + trends.
 HISTOGRAM: distribution buckets (request_latency_seconds)
   → percentiles computable: p50/p95/p99 ✓
 SUMMARY: client-side quantiles (limited aggregation across
   instances — histograms usually preferred platform-wide)

 latency example done right:
   histogram: http_request_duration_seconds{route,method}
   alert on: histogram_quantile(0.99, rate(...[5m])) 

What to Instrument First

 the USE/RED canon:

 RED for SERVICES (traffic-facing):
   Rate    requests/sec per endpoint-class
   Errors  failed/sec (status AND business failures!)
   Duration latency histogram per endpoint

 USE for RESOURCES:
   Utilization, Saturation, Errors per resource:
   cpu/mem/conn-pools/thread-pools/queues/disk-io

 plus BUSINESS counters (the ones that catch what tech
 metrics miss): orders_created, checkout_completed,
 payment_declines — anomalies here = real incidents even
 when all systems show green.

 priority order when starting: RED first, business counters
 second, USE as capacity evidence accumulates.

Cardinality: The Silent Killer

 every label VALUE multiplies stored series:

 http_requests{user_id=912}        ← 10M users = 10M series ✗✗
 http_requests{route=/orders}      ← bounded routes ✓

 rules that keep platforms alive:
 □ NEVER label with: user ids, request ids, emails, raw URLs
   (normalize /orders/{id} → route template /orders/:id)
 □ budget: hundreds of series per service, not thousands
 □ high-cardinality NEEDS go to traces/logs with exemplar links,
   never into metric labels
 cardinality explosions have taken down monitoring backends
 during incidents — exactly when you need them most.

Instrumentation Quality Checklist

 □ CONSISTENT naming: namespace_subsystem_unit convention
   (http_server_duration_ms); lint it in CI
 □ EVERY counter paired with its failure twin:
   requests_total + errors_total (or status-code labels)
 □ HISTOGRAM buckets sized to your SLOs: bucket boundary AT
   the SLO threshold exists (or p99-vs-SLO math lies)
 □ BUSINESS events counted at commit points, not attempts
 □ SELF-METRICS: instrument the instrumentation (export
   queue depths, dropped samples) — observability observes itself

Interview Framing

“Instrument this new checkout service” scored shape: RED set enumerated per endpoint with histogram-for-latency reasoning, business counters included unprompted (checkout_completed vs attempted!), cardinality rules quoted with the URL-normalization example, bucket-boundary-at-SLO subtlety named. Instrumentation questions grade measurement LITERACY — the cardinality answer alone separates practitioners from tutorial-followers.