Correlation IDs

The Needle-in-Haystack Problem

 user reports: "my order vanished."
 order touches: gateway → orders → payments → inventory → email

 without correlation: grep 6 services' logs by timestamp+email
 guesswork. with it: one id retrieves the entire journey:

 [gateway]  cid=abc-123 POST /checkout
 [orders]   cid=abc-123 created order 88
 [payments] cid=abc-123 charged $89.90 ✓
 [inventory]cid=abc-123 reserved sku-55 ✗ timeout!
 [email]    cid=abc-123 confirmation sent

 CORRELATION ID: a unique tag assigned at flow entry,
 propagated EVERYWHERE that work goes.

Propagation Mechanics

 SYNCHRONOUS hops:
   HTTP header X-Correlation-ID / traceparent — middleware
   injects on entry, forwards downstream, logs everywhere.
   MISSING incoming id? GENERATE one (never log untagged).

 ASYNCHRONOUS hops (the ones teams forget):
   message headers carry the id; consumers extract + continue.
   saga/event chains keep the ORIGINAL flow-id alive across
   hours of async processing.

 SCHEDULED/background work:
   synthesize ids per execution; include job-name in context.

 implementation home: middleware/interceptors ONCE per service,
 not developer memory. framework-enforced beats convention-hoped.

Correlation ID vs Trace ID

	Trace ID	Correlation ID
scope	one request’s synchronous path	whole BUSINESS action
survives queues?	typically no (new spans)	yes — designed to
tooling	tracing systems	log search / support tickets
granularity	milliseconds precision	end-to-end narrative

 mature systems carry BOTH:
 trace_id for latency forensics within sync paths;
 correlation_id stitching the multi-day, multi-system story
 ("where did THIS refund request go after the retry queue?").

 support-facing bonus: return/show correlation ids to users
 and CS agents — "give me the code from your error screen"
 turns tickets into instant investigations.

Operational Discipline

 □ MANDATORY at edges: gateways generate if absent;
   services reject-or-tag missing contexts (lint via tests)
 □ INDEXED in log stores as first-class field
 □ PROPAGATED through: HTTP, messaging, grpc metadata,
   scheduled jobs, even batch scripts touching prod
 □ SAMPLED-SAFE: correlation works regardless of trace
   sampling decisions (ids exist even when spans don't)
 □ RETENTION aligned: the id is only as good as the shortest-
   retained store in its chain

Interview Framing

“A payment succeeded but the order shows failed — investigate” scored shape: correlation-id propagation design spanning sync AND async hops, middleware-enforced generation, both-ids distinction explained, support-facing usage mentioned. This question appears constantly because distributed debugging IS correlation discipline — candidates who start grepping timestamps fail it quietly.