Isolation and Blast Radius

Blast Radius Defined

 BLAST RADIUS = the set of users/tenants/features harmed when
                a component fails.

 goal of isolation design: make the answer SMALL and KNOWN,
 not "depends what breaks."

 unisolated:  one shared DB pool → one runaway query → everyone down
 isolated:    per-tenant pools     → same query harms one tenant

Isolation Axes

 choose WHERE to put walls:

 by TENANT      enterprise A can't hurt enterprise B
                  (per-tenant queues, pools, or cells)
 by FEATURE     checkout failure ≠ search failure
                  (separate services + resources)
 by DEPENDENCY  payment-PSP outage ≠ catalog outage
                  (bulkheads per upstream)
 by REGION      one region's config error stays regional
                  (cells / multi-region)
 by LOAD CLASS  interactive traffic shielded from batch jobs
                  (separate pools + priorities)

Mechanisms That Create Distance

Mechanism	Isolates
Bulkheads (pools per dependency)	Slow upstream from starving everything
Rate limits per tenant	Noisy neighbor from shared capacity
Queue-per-consumer-class	Batch backlog from interactive path
Separate credentials/quotas downstream	One service exhausting shared API limits
Circuit breakers	Cascading failure across hops
Cell partitioning	Infrastructure-level incidents

The Shared-Resource Audit

 most blast-radius surprises hide in SHARED things nobody listed:

 - one database connection pool for all dependencies
   → slow PSP holds connections → checkout AND profile fail
 - one Kafka topic for all event types
   → poison message blocks unrelated consumers
 - one API key for a third party across all features
   → quota exhausted by feature X starves feature Y
 - one cache cluster, no key-prefix quotas
   → tenant flood evicts everyone's entries

 audit question for each: "what ELSE dies if this saturates?"
 if the answer is "everything," that's the next wall to build.

Measuring the Radius

 make blast radius a TESTED number, not a hope:

 - fault injection: kill a dependency in staging; count affected flows
 - game days: deliberately degrade one cell/pool in production
   with rollback armed
 - per-tenant dashboards: detect noisy neighbors in minutes, not tickets
 
 target statement worth writing down:
   "any single component failure affects ≤ 5% of traffic"
   then engineer and verify toward it

The Cost Side

 isolation trades efficiency:
 - N small pools = more idle capacity than one big pool
 - per-tenant infrastructure = minimum cost per tiny tenant
 - more configs = more places to misconfigure

 ladder of expense:
   pooled + rate-limited (cheap) →
   pooled + bulkheads (moderate) →
   dedicated instances per tenant (expensive) →
   full cells (most expensive)

 match the wall to the consequence: isolate what KILLS you,
 rate-limit what merely annoys.

Interview Framing

Multi-tenant designs earn points here fast: name the shared resources, assign walls (“per-tenant rate limits at the gateway, separate worker pools for exports vs interactive”), state the target radius numerically. The phrase “blast radius” used precisely — with a percentage attached — is a strong seniority signal in any design review.