Sampling Strategies

Why Sampling Is Non-Negotiable

 full tracing at scale = storage/bandwidth death:

 10k req/s × ~15 spans × ~1KB = 150MB/s of trace data.
 keeping it all costs more than most services' compute.

 sampling keeps INSIGHT while discarding VOLUME —
 the craft is deciding WHAT to keep, because the wrong
 strategy hides exactly the traces you need at 3am.

Head vs Tail

 HEAD SAMPLING (decide at trace start):
   coin flip at entry; whole trace kept-or-dropped uniformly
   ✓ cheap, simple, context-consistent (flag propagates)
   ✗ blind: errors/slow traces dropped at the same rate!
     a 1% sample throws away 99% of your incident evidence

 TAIL SAMPLING (decide at trace END, at collector):
   buffer whole traces; apply POLICY:
     keep ALL errors
     keep latency > threshold (p99-hunters)
     keep rate-limited baseline of successes
   ✓ keeps the interesting 100%
   ✗ buffering cost + complexity + cross-service coordination
     (collector must see complete traces)

 modern default: TAIL with policy rules, sized honestly;
 head-only only where infrastructure truly can't tail.

Policy Design That Works

 a production-grade tail policy set:

 □ errors:        100% kept (non-negotiable evidence)
 □ slow:          > p95-threshold kept (latency forensics)
 □ baseline:      success sampled to fixed RPS per service
                  (guarantees steady-state visibility)
 □ special:       baggage-flagged debug sessions 100% ✓
 □ new-deploy:    temporarily raise rates during rollouts

 guardrails on the policies themselves:
 □ GLOBAL RATE LIMITS after policies (a service error-storm
   must not flood the backend even though "errors=keep"!)
 □ per-policy budgets reviewed quarterly against actual value
 □ DROPPED-COUNT metrics exported: you must know what you
   didn't see

Strategy	Cost	Incident readiness
no sampling	extreme	perfect until bill arrives
head uniform	low	blind during incidents
tail policy + limits	medium	excellent
adaptive (below)	higher	best

Adaptive and Emerging Approaches

 ADAPTIVE: controllers watch traffic/error mix and tune
   per-route sample rates automatically (open-telemetry's
   tail-sampling + processors, vendor systems).
 
 EXEMPLARS as force-multiplier:
   even 1%-sampled traces gain power when METRIC points
   LINK to their matching trace exemplars — dashboards
   surface relevant samples on demand instead of search.

 the honest framing for interviews: sampling trades
 completeness for viability; policies bias retention toward
 failure/latency because those are what investigations need.

Interview Framing

“Tracing is costing us $200k/month” scored shape: volume math motivating sampling, head-vs-tail mechanics with the blindness flaw named, concrete policy list (errors-100%, slow-kept, rate-limited baseline), global-rate-limit guardrail, exemplar linkage mentioned. Sampling questions test whether you treat observability as ENGINEERED ECONOMICS — budgets and policies over defaults.