Statistics & A/B Testing

A/B testing is the #1 topic for Applied Scientist roles — expect 1-2 full rounds. You must be able to design an experiment from scratch (sample size, randomization, duration), interpret results correctly (p-value, confidence intervals, power), and handle complications (novelty effects, network effects, multiple testing). This is tested at Amazon, Meta, Google, LinkedIn, Airbnb.

Probability distributionsHypothesis testingBasic statisticsConfidence intervalsA/B TestingSample SizeCUPEDMultiple TestingStatistical Power

Core Theory

1Hypothesis testing structure: H₀: μ_treatment = μ_control (no effect). H₁: μ_treatment ≠ μ_control. Test statistic: z = (x̄_T - x̄_C) / √(σ²/n_T + σ²/n_C). Under H₀, z ~ N(0,1). Reject if |z| > z_{α/2}. A p-value of 0.03 means: if the null is true, 3% chance of observing data this extreme. It is NOT the probability the null is true.
2Sample size formula: n = 2σ²(z_{α/2} + z_β)² / δ² where δ = minimum detectable effect (MDE), σ = standard deviation of the metric, α = significance level (Type I error rate), β = Type II error rate, power = 1-β. Know this formula and be able to compute n given α=0.05, power=0.80, MDE, σ.
3CUPED (Controlled-experiment Using Pre-Experiment Data): variance reduction technique. Use a pre-experiment covariate X (same metric measured before the experiment) to reduce noise. Adjusted metric: Y_cuped = Y - θ(X - E[X]) where θ = Cov(Y,X)/Var(X). Reduces variance by 1 - ρ²(Y,X). If pre-experiment CTR is highly correlated with experiment CTR (ρ≈0.7), CUPED reduces variance by 51%, allowing you to run a shorter experiment or detect smaller effects.
4Multiple testing: if you test 20 metrics at α=0.05, you expect 1 false positive by chance. Methods: (1) Bonferroni correction — divide α by number of tests (α/m). Very conservative. (2) Benjamini-Hochberg FDR — controls expected fraction of false discoveries among rejections. Less conservative. (3) Pre-specify primary and secondary metrics before looking at data — only the primary metric is used for the ship/no-ship decision.
5Novelty effects: users interact differently with new features just because they're new (curiosity effect) or because they resist change. Detection: plot the treatment effect over time. A genuine effect is stable; a novelty effect decays. Run experiments for at least 2 weeks and check if the day-1 effect matches the day-14 effect. Primacy effects: users who dislike change initially underperform but improve over time.
6Network effects and interference: in social networks, a user's behavior can affect others in a different treatment group (spillover). Treatment users sharing content with control users violates the Stable Unit Treatment Value Assumption (SUTVA). Solutions: cluster randomization (randomize by user cluster/community), ego-network analysis, switchback experiments (alternating treatment/control over time for a single unit).
7Confidence interval interpretation: a 95% CI means that if you repeated the experiment many times, 95% of the constructed intervals would contain the true parameter. It does NOT mean there is 95% probability the true value is in this specific interval (that requires Bayesian credible intervals). This distinction is tested directly in Applied Scientist interviews.
8Sequential testing: instead of waiting for a fixed sample size, use sequential probability ratio test (SPRT) or alpha-spending functions (O'Brien-Fleming) to peek at results early without inflating Type I error. Used at Airbnb and LinkedIn to enable early stopping when effects are clearly positive or negative.

Key Formulas

01Sample size:

n = 2\sigma^2(z_{\alpha/2} + z_\beta)^2 / \delta^2

02Test statistic:

z = (\bar{x}_T - \bar{x}_C) / \sqrt{\sigma^2(1/n_T + 1/n_C)}

03CUPED:

Y_{adj} = Y - \hat{\theta}(X - \bar{X})

\hat{\theta} = Cov(Y,X)/Var(X)

04Variance reduction:

Var(Y_{adj}) = Var(Y)(1 - \rho^2_{Y,X})

05Bonferroni:

\alpha_{adj} = \alpha / m

(m = number of tests)

06Power:

1 - \beta = \Phi(|\delta|\sqrt{n/2}/\sigma - z_{\alpha/2})

python

1	import numpy as np
2	from scipy import stats
3
4	np.random.seed(42)
5
6	# ─── 1. Sample size calculator ─────────────────────────────────────────────────
7	def sample_size(baseline_rate: float, mde_relative: float,
8	alpha: float = 0.05, power: float = 0.80) -> int:
9	"""
10	Calculate per-arm sample size for a proportion metric (e.g., CTR).
11	mde_relative: minimum detectable effect as fraction of baseline
12	e.g. 0.05 = detect 5% relative lift on a 10% CTR
13	"""
14	p1 = baseline_rate
15	p2 = baseline_rate * (1 + mde_relative)
16	sigma_sq = (p1(1-p1) + p2(1-p2)) / 2 # pooled variance
17	delta = p2 - p1
18	z_a = stats.norm.ppf(1 - alpha/2) # 1.96 for alpha=0.05
19	z_b = stats.norm.ppf(power) # 0.84 for power=0.80
20	n = 2 * sigma_sq * (z_a + z_b)2 / delta2
21	return int(np.ceil(n))
22
23	baseline_ctr = 0.10 # 10% click-through rate
24	mde = 0.05 # detect 5% relative lift (10% → 10.5% CTR)
25
26	n = sample_size(baseline_ctr, mde)
27	print(f"Sample size per arm: {n:,}")
28	print(f"Total users needed: {2*n:,}")
29	print(f"Days needed (10k DAU): {2*n/10_000:.0f} days")
30
31	# ─── 2. CUPED variance reduction ──────────────────────────────────────────────
32	n_users = 10_000
33	true_effect = 0.005 # 0.5pp absolute lift
34
35	# Simulate users with pre-experiment CTR
36	pre_exp_ctr = np.random.beta(2, 18, n_users) # ~10% mean
37	noise = np.random.randn(n_users) * 0.03
38
39	control = pre_exp_ctr + noise
40	treatment = pre_exp_ctr + noise + true_effect * np.random.choice([0, 1], n_users, p=[0.5, 0.5])
41	is_treatment = np.random.choice([0, 1], n_users, p=[0.5, 0.5]).astype(bool)
42
43	Y_c = control[~is_treatment]
44	Y_t = treatment[is_treatment]
45	X_c = pre_exp_ctr[~is_treatment] # pre-experiment covariate
46	X_t = pre_exp_ctr[is_treatment]
47
48	# CUPED adjustment
49	all_Y = np.concatenate([Y_c, Y_t])
50	all_X = np.concatenate([X_c, X_t])
51	theta = np.cov(all_Y, all_X)[0, 1] / np.var(all_X)
52
53	Y_c_adj = Y_c - theta * (X_c - all_X.mean())
54	Y_t_adj = Y_t - theta * (X_t - all_X.mean())
55
56	# Compare variance
57	print(f"\nOriginal variance: {np.var(Y_t) + np.var(Y_c):.6f}")
58	print(f"CUPED variance: {np.var(Y_t_adj) + np.var(Y_c_adj):.6f}")
59	corr = np.corrcoef(all_Y, all_X)[0, 1]
60	print(f"Variance reduction: {(1 - corr*2)100:.1f}% (rho={corr:.2f})")
61
62	# ─── 3. Multiple testing: false positive inflation ────────────────────────────
63	n_experiments = 20
64	alpha = 0.05
65	n_per_arm = 500
66
67	false_positives = 0
68	for _ in range(10_000):
69	# Simulate 20 metrics with no true effect
70	control = np.random.randn(n_per_arm)
71	treatment = np.random.randn(n_per_arm)
72	p_values = [stats.ttest_ind(
73	control + np.random.randn(n_per_arm) * 0.1,
74	treatment + np.random.randn(n_per_arm) * 0.1
75	).pvalue for _ in range(n_experiments)]
76	if any(p < alpha for p in p_values):
77	false_positives += 1
78
79	print(f"\nFalse positive rate (1 metric): {alpha:.3f}")
80	print(f"False positive rate (20 metrics): {false_positives/10_000:.3f}")
81	print(f"Expected: 1 - (1-{alpha})^20 = {1-(1-alpha)**20:.3f}")

Three critical A/B testing computations: (1) sample size calculation from first principles — the formula gives ~31k users per arm to detect a 5% relative lift on 10% CTR; (2) CUPED variance reduction showing that pre-experiment correlation ρ reduces required sample size by (1-ρ²) × 100%; (3) multiple testing simulation proving that testing 20 metrics at α=0.05 inflates false positive rate from 5% to ~64%.

Worked Interview Problems

3 problems

Problem

The product team wants to test a new homepage design. Current conversion rate is 3%. They want to detect a 10% relative improvement (3% → 3.3%). How do you design the experiment?

Solution

Define metrics. Primary metric: conversion rate (3-day purchase after visit). Guardrail metrics: latency P99 (must not degrade), bounce rate (must not increase >2%), customer satisfaction score.

Sample size: σ² ≈ p(1-p) = 0.03×0.97 = 0.0291. δ = 0.003 (absolute). z_α/2 = 1.96, z_β = 0.84. n = 2×0.0291×(1.96+0.84)²/0.003² = 2×0.0291×7.84/0.000009 ≈ 50,667 per arm. Need ~100k total users.

Duration: if site gets 10k unique visitors/day, need 10 days minimum. Run for 14 days to cover weekly seasonality (behavior differs weekday vs weekend). Never stop early based on interim results (use sequential testing if you need to peek).

Randomization: randomize at user level (consistent experience across sessions). Use a hash of user_id + experiment_id for deterministic assignment without a lookup table.

Novelty effects: plot conversion rate by day for each group. If treatment shows a spike on day 1-2 then drops, this is a novelty effect. Use the stabilized effect (day 7-14) as the estimate, not the overall mean.

Analysis: z-test for proportions. Report: absolute lift, relative lift, 95% CI, p-value, power (was the experiment adequately powered for the observed effect size?).

Answer

Sample size: ~50k/arm, 100k total. Duration: 14 days minimum (cover weekly seasonality). Randomize at user level. Primary metric: 3-day conversion rate. Guardrails: latency, bounce rate. Check for novelty effects by plotting treatment effect over time.

Common Mistakes

Peeking at experiment results and stopping early when p<0.05. This massively inflates false positive rates — if you check daily, your effective α can reach 30%+ by day 30. Use sequential testing methods (SPRT) if you need to peek, or pre-commit to a fixed duration.

Using the same users for both parameter tuning and final evaluation. If you ran 3 experiments to pick the best homepage variant, each at α=0.05, the probability that the winner is a false positive is much higher than 5%.

Misinterpreting the confidence interval as a probability statement: 'there is 95% probability the true effect is in [5%, 15%].' A CI is a procedure: 95% of intervals constructed this way contain the true value. For the probability interpretation, use Bayesian credible intervals.

Not accounting for network effects in social platform experiments. If treatment users share content with control users (or vice versa), the independence assumption is violated. The measured effect will be an underestimate (interference dilutes the treatment effect).

Using session-level randomization when users have multiple sessions. Session-level gives each session independent treatment assignment, so the same user might be in both treatment and control across different sessions — this is a contaminated experiment. Use user-level randomization.

Back to ML Hub

Feature Engineering & Data Quality Causal Inference