A/B Testing

The Experiment Shape

 split traffic between variants; measure outcome differences:

 [users]──hash(user_id)──► bucket A: control (current)
                        └► bucket B: treatment (new checkout)

 same exposure conditions, random assignment, ONE variable:
   conversion A: 2.10%   B: 2.17%   ← is that real?

 the machinery overlaps canary/flags but the QUESTION differs:
 canary asks "is v2 broken?"      (operational health)
 A/B asks "is v2 BETTER?"         (product impact)

 different gates, different statistics, different owners.

The Statistics You Cannot Skip

 □ SAMPLE SIZE BEFORE STARTING: minimum detectable effect ×
   baseline rate → required n per arm. "we'll watch and see"
   = peeking = lying to yourselves with noise.
     example math: baseline 2%, +5% relative lift to detect,
   95% confidence → ~200k users PER ARM. small sites: weeks.
 □ DON'T PEEK-AND-STOP: early "wins" are mostly variance;
   pre-commit to duration OR use sequential-testing methods
 □ ONE METRIC DECIDES (OEC): overall evaluation criterion
   chosen BEFORE launch; guardrail metrics constrain harm
   (latency, error budgets, revenue floors)
 □ NOVELTY EFFECTS: early lift decays as novelty fades —
   run full business cycles (whole weeks) minimum.

Assignment Infrastructure

 consistent bucketing machinery:

 user_id ──salted hash──► [0-99] buckets
   experiment X uses buckets 20-29: 90 control / 10 treatment

 requirements that bite when ignored:
 □ STICKINESS: same user, same variant, entire experiment
 □ ISOLATION: concurrent experiments must not interact —
   layered/nested bucketing domains prevent collision
 □ NEUTRAL SALT per experiment (avoid correlated assignments)
 □ LOG THE ASSIGNMENT with every event — analysis dies
   without exposure records ✓

Pitfall	Consequence
peeking early	false winners shipped
metric soup post-hoc	”significance” found somewhere always
assignment leaks (bot traffic)	polluted arms
cross-experiment interference	uninterpretable results
UI flicker on variant load	bias against treatment

The Platform Reality

 at scale this becomes infrastructure:

 - assignment service (or SDK-local hashing) at edge speed
 - event pipeline joining EXPOSURE + OUTCOME streams
 - analysis tooling with sequential tests built in
 - governance: experiment registry, review of metrics/OEC,
   automatic shutdown on guardrail breach

 small teams: a flag system + discipline + a spreadsheet-era
 stats book covers 80% of value. don't build Netflix's
 platform for your first test.

Interview Framing

“Should we redesign the search results page?” scored shape: refuse opinion-war framing, propose experiment design (one OEC, sample-size math shown, duration commitment), assignment-infrastructure needs named, pitfalls volunteered (peeking, novelty). A/B questions test whether you’d let EVIDENCE decide product questions — and whether you know how easily bad experiments lie.