Canary Routing

The Mesh Canary Stack

 three cooperating features:

 1. SUBSETS: label pods by version; discovery tracks groups:
    reviews: {v1: 8 pods, v2: 2 pods}

 2. WEIGHTED ROUTING: percentages between subsets
    (traffic-shifting lesson's primitive)

 3. MIRRORING (shadow traffic): send COPY to v2,
    fire-and-forget response:

 [client]──►[proxy]──90%──►[v1]──►real response──►client
                 └─copy─►[v2]──(response discarded)

 v2 experiences REAL production shapes without user risk.
 the dress rehearsal before tickets are sold.

The Automated Canary Loop

 manual weight-flipping doesn't scale across hundreds of
 services. the pipeline:

 deploy v2 → mirror phase → compare responses/metrics
   → 1% real → GATE: error-rate delta < threshold?
   → 5% → gate → 25% → gate → 100% → v1 retired

 gates computed from MESH TELEMETRY (uniform per-version
 metrics make this tractable):
 - request success rate per subset
 - p50/p95/p99 latency deltas  
 - saturation signals on v2 pods
 - business counters if instrumented

 tools in this space: flagger-class controllers automate
 the loop; Argo Rollouts integrates similarly. concept >
 tool choice.

Mirror-Mode Caveats

 shadow traffic is powerful but not free of traps:

 □ IDEMPOTENCY REQUIRED on mirrored paths: v2 receives REAL
   writes twice-ish (shadowed + later real). side effects must
   be safe or discarded via header-stripping at test sinks.
   common practice: strip/flag shadow requests so v2's
   downstream writes no-op or hit sandboxes.
 □ DOUBLE LOAD on shared downstreams during mirrors —
   capacity-check dependencies first
 □ RESPONSE COMPARISON needs normalization (timestamps,
   ids) or every diff looks like a bug
 □ ASYNC/side-effectful flows can't be fully validated by
   response-diffing alone — pair with staged real traffic.

Phase	Risk	Gate
Mirror	none to users	diff-rate < threshold
1%	minimal	auto-metrics pass
ramp	bounded	each step gated
100%	full	post-promo watch window

Interview Framing

“Ship a high-risk pricing-engine rewrite” scored shape: full stack proposed (mirror→gated ladder), shadow-mode caveats shown (idempotency/double-load), gate metrics named from mesh telemetry, rollback-as-weight-reversion noted, automation mentioned for fleet scale. Canary-routing questions reward demonstrating the FULL machinery including its traps — anyone can say ‘canary’; operators know what shadows break.