Distributed Tracing

The Anatomy

 a TRACE = tree of SPANs, one per unit of work:

 trace: checkout-abc (total 620ms)
 ├─ span: gateway          5ms
 ├─ span: orders         180ms
 │   ├─ span: db.query   150ms   ← the story lives here!
 │   └─ span: cache.get    2ms
 ├─ span: payments       400ms  ← THE culprit visible instantly
 │   └─ span: http.bank-call 395ms
 └─ span: email(async)    30ms

 each span carries: service, operation, start/duration,
 status, tags (http.status, db.statement class), and its
 parent — reconstructing causality across processes.

What Traces Answer That Nothing Else Can

 □ WHERE did the 800ms go? (the waterfall shows the hop)
 □ WHICH hop failed in a fan-out of nine calls?
 □ FAN-OUT exposure: this endpoint calls 14 downstreams?! 
   (architectural surprises revealed daily)
 □ N+1 patterns: 200 identical db spans inside one request ✓
 □ cross-service attribution: whose timeout caused whose error?

 logs say SOMETHING failed; metrics say failure RATE;
 only traces show the PATH and the TIME BUDGET breakdown.

Instrumentation Layers

 zero-code layer:
   mesh sidecars auto-create proxy spans + propagate context
   → inter-service skeleton for free

 framework/auto-instrumentation (OTel):
   http servers/clients, db drivers, queues instrumented via
   libraries/config — covers 80% of real spans without code

 manual spans for BUSINESS meaning:
   span("price-calculation") / attributes(order_total=…, tier=gold)
   → the difference between debugging and understanding

 adoption order: auto-instrument fleet-wide first,
 add meaningful manual spans where waterfalls stay confusing.

The Practical Discipline

Concern	Practice
context loss at async boundaries	propagate headers INTO message payloads; consumers resume traces
missing spans	verify propagation FIRST (most “tracing bugs” are header drops)
useless tag soup	standardize semantic conventions (http/db/messaging)
backend cost	sampling strategy is mandatory (own lesson)
clock skew across hosts	span timing tolerances; don’t over-trust microsecond ordering

 the async-boundary gap is the classic implementation wound:
 kafka consumer starting FRESH traces because nobody copied
 headers into message properties — half your latency story
 silently vanishes into disconnected trace fragments.

Interview Framing

“p99 doubled after a new release but no service admits it” scored shape: trace-waterfall walkthrough as THE diagnostic (find the fat span), instrumentation-layer plan (mesh+auto+manual-business-spans), async-propagation gotcha volunteered, sampling acknowledged with pointer to strategy. Tracing questions test causal-reasoning fluency — walk the waterfall out loud and the interview grades itself.