Logs, Metrics, Traces

The Three Questions

 METRICS answer "HOW MUCH / IS IT HEALTHY?":
   numeric time-series, cheap at scale, alertable:
     request_rate, error_ratio, p99_latency, queue_depth

 LOGS answer "WHAT EXACTLY HAPPENED?":
   discrete event records with rich detail:
     the error's stack trace, the rejected payload, audit trail

 TRACES answer "WHERE DID TIME/FAILURE GO?":
   causal path through services:
     gateway(5ms)→orders(120ms→db 90ms)→payments(300ms!) 

 [alert fires on METRIC] → [TRACE shows the slow hop]
                         → [LOGS show the exact error]
 the pillars form an INVESTIGATION PIPELINE, not a menu.

The Cost/Detail Tradeoff

Pillar	Volume	Cost	Retention reality
metrics	tiny (aggregates)	low	months-years cheap
traces	sampled subset	medium	days-weeks
logs	every event	HIGH	days-weeks typically

 logs dominate spend in most orgs — a single chatty service
 can out-cost everything else. hence the discipline trio:
 □ log LEVELS enforced (debug off in prod by default)
 □ structured + sampled where volume demands
 □ metrics/traces carry you to the neighborhood;
   targeted logs finish the story. don't log everything forever.

How They Interlock

 the linkage fields that make investigation flow:

 trace_id IN LOGS:      pull all logs for one request's journey
 trace_id IN METRICS:   exemplars — metric point links to example trace
 correlation_id:        business-level stitching across async hops

 incident flow in practice:
 p99 spike alert (metric)
 → open trace exemplar → payments hop +280ms
 → grep logs by trace_id → connection-pool timeout logged
 → fix; dashboard confirms recovery (metric) ✓

Building the Stack

 pragmatic component choices per pillar:

 metrics: prometheus/mimir-class + grafana dashboards
 traces:  OTLP → jaeger/tempo-class backends, head sampling
 logs:    structured JSON → loki/elastic-class stores

 open-telemetry as the INSTRUMENTATION LAYER unifies all three:
 one SDK, one config, vendor-swappable backends. adopt OTel
 semantics even if starting with one pillar only.

 anti-patterns to refuse:
 ✗ three disconnected tools with no id linking
 ✗ logging INSIDE loops at info level ("processing item...")
 ✗ tracing without sampling strategy (own lesson)

Interview Framing

“Design observability for this new service” scored shape: three-pillar division stated with their QUESTIONS, cost asymmetry acknowledged (logs!), interlock via trace-id demonstrated through an investigation walkthrough, OTel named as instrumentation standard. This question is the field’s entry exam — graded on pipeline-thinking rather than tool-name-dropping.