Menu

Earn Premium with Referrals

Invite your friends and earn Premium rewards through our referral program.

See how it works and start inviting friends.

Mesh Timeouts
HLD

Mesh Timeouts

Deadline discipline as declarative config — route timeouts, per-try budgets, and propagation gaps.

Timeouts Everywhere by Default

 mesh gives every hop an EXPLICIT timeout — the "no call
 without a limit" rule enforced fleet-wide:

 route-level:
   timeout: 3s          ← whole-request envelope at THIS hop
   retries.perTryTimeout: 1s  ← each attempt's slice

 defaults matter: many meshes ship conservative defaults
 (istio's historic 15s-ish) that silently shape your SLOs.
 AUDIT them like code, don't inherit them blindly.

The Budget Chain in a Mesh World

 same arithmetic as always, now distributed across proxies:

 user budget 1000ms
   gateway→orders:    timeout 500ms (retries: 2×200ms)
   orders→payments:   timeout 300ms
   payments→fraud:    timeout 100ms ← tightest leaf

 rules re-verified per hop:
 □ parent > child sums (or parent cancels children)
 □ per-try ≤ overall with room for actual retries
 □ DB/cache calls inside services still need THEIR limits —
   mesh covers service-to-service; app-side calls remain yours

 DEADLINE PROPAGATION GAP to know about:
 meshes don't automatically shrink downstream timeouts by
 elapsed upstream time unless headers carry deadlines and
 both ends honor them. check YOUR stack; assume gaps otherwise.

Timeout Failure Modes Specific to Meshes

 □ DOUBLE-WRAP confusion: app sets 5s client timeout AND
   mesh route says 15s → effective 5s; dashboards show
   "mesh timeouts" never firing → false confidence.
   ONE owner per limit layer; document who owns which.

 □ TIMEOUT vs RETRY interaction misconfigured:
   timeout 2s, retry ×3, per-try unset → one slow attempt
   eats everything; retries never happen (see retries lesson)

 □ LONG-POLL/streaming endpoints killed by blanket L7
   timeouts → exempt via explicit long-timeout routes or
   L4 pass-through for streams. enumerate these endpoints!

 □ IDLE timeouts on connections (TCP level) vs REQUEST
   timeouts conflated in reviews — different dials!
Endpoint classTimeout posture
Interactive readstight (p99×margin)
Checkout writesbounded + idempotent retries
Reports/exportsasync-ify; no giant sync timeouts
Websockets/streamsidle-based, exempt from request caps

Interview Framing

“Random 504s from the mesh after our p99 got worse” scored shape: budget-chain audit approach (parent/child arithmetic), double-wrap ownership check first, retry/per-try interaction verified second, streaming-endpoint exemptions third. Mesh-timeout incidents are almost always CONFIG INTERACTION bugs — demonstrating the audit ladder is the competence signal.

My Private Notes

Notes are auto-saved locally to this device.