Circuit Breakers in Microservices

The Graph Problem

 microservices = dependency GRAPHS; failures propagate:

 [frontend]──►[orders]──►[payments]     (down!)
                     └─►[inventory]
                     
 payments dying hangs orders' threads → orders slows →
 frontend's calls to orders hang → cascade (cascading-
 failures lesson) UNLESS every hop breaks the chain.

 MICROSERVICE-SPECIFIC twist: breakers must exist at
 EVERY hop, per-dependency — not just at the edge.

Placement and Scoping

 WHERE: wrap EVERY remote call:
   service-to-service, service-to-DB-driver-level,
   service-to-cache, service-to-third-party.

 SCOPING rules that matter:
 □ PER-DEPENDENCY: orders→payments breaker ≠ orders→inventory
   one sick dependency must never trip healthy ones
 □ PER-ENDPOINT-CLASS within a dependency when behavior differs
   (reads vs writes; cheap vs heavy)
 □ SHARED STATE per instance is fine (fleet-wide coordination
   usually unnecessary complexity)

 implementation homes:
   libraries (resilience4j-era), sidecars/mesh policy,
   or generated clients with resilience baked in.

The Microservices Failure Choreography

 breaker trips on orders→payments. now what? THE PLAN:

 [orders]──(OPEN breaker)──►[payments ✗]
     │
     ├─ fallback: mark order PAYMENT_PENDING,
     │            queue for async retry ✓ (order survives!)
     ├─ metric: breaker-state exported; dashboard lights up
     └─ downstream awareness: frontend shows honest state

 contrast with NO plan: exception bubbles → order lost →
 support tickets → revenue leak.

 rule: every breaker ships WITH its fallback design
 (fallbacks lesson's taxonomy applies directly).
 tripping without a fallback converts slow failure
 into fast failure — better, but still failure.

Tuning in a Service Graph

Concern	Guidance
Thresholds	per-dependency baselines; no global constants
Retry+breaker interaction	retries INSIDE closed state only; open = instant fallback
Half-open probes	tiny; respect downstream recovery time
Cascading trips	expected during real incidents — shedding upstream keeps cores alive
Flapping	hysteresis + longer cooldowns; alert on trip-rate

 observability requirement:
 export breaker STATE transitions as events
 (name, from→to, reason) — incident timelines depend on
 reconstructing exactly which breakers opened when.

Interview Framing

“Payments service degrades; walk me through your system’s response” scored shape: per-hop breaker placement drawn on the dependency graph, OPEN-state choreography with concrete fallback (pending+queue, not error-to-user), scoping rules stated (per-dependency!), retry/breaker interaction handled correctly, state-transition observability named. This question tests whether resilience patterns live in your design as a COORDINATED system or as buzzwords sprinkled per box.