Applied Scientist
Causal Inference
Causal inference is essential when you can't randomize. Applied Scientists at tech companies face this constantly: 'Did this feature launch actually improve retention, or did it just coincide with a seasonal uplift?' You must know the potential outcomes framework, confounding, and the four main non-experimental methods.
Core Theory
- 1Potential outcomes framework (Rubin): each unit i has two potential outcomes: Y_i(1) under treatment, Y_i(0) under control. The individual treatment effect ฯ_i = Y_i(1) - Y_i(0) is never observed (fundamental problem of causal inference). ATE = E[Y(1) - Y(0)], ATT = E[Y(1) - Y(0) | T=1]. SUTVA: each unit's outcome depends only on its own treatment (no interference).
- 2Confounding: a variable C is a confounder if it affects both treatment assignment T and outcome Y. Classic example: healthier people are more likely to exercise (T) AND have better health outcomes (Y). Naive comparison overstates the exercise benefit. Causal graphs (DAGs) make confounders explicit: C โ T and C โ Y. Block all backdoor paths from T to Y to get the causal effect.
- 3Propensity score methods: propensity score e(x) = P(T=1|X) โ probability of treatment given covariates. Under ignorability (no unmeasured confounders), conditioning on the propensity score removes confounding. Methods: (1) Matching โ for each treated unit, find control unit with similar e(x). (2) IPW (Inverse Probability Weighting) โ weight each unit by 1/e(x) for treated, 1/(1-e(x)) for controls. (3) Doubly robust โ combines propensity weighting with outcome regression.
- 4Difference-in-Differences (DiD): estimate causal effect by comparing the change over time in the treatment group to the change in the control group. Parallel trends assumption: in the absence of treatment, both groups would have had the same trend. DiD estimate: ฯ_DiD = (Y_T,post - Y_T,pre) - (Y_C,post - Y_C,pre). Can include unit fixed effects to control for time-invariant confounders.
- 5Regression Discontinuity (RD): when treatment is assigned based on a threshold (e.g., get scholarship if score โฅ 75), units just below and above the threshold are similar except for treatment. The causal effect is estimated by comparing outcomes just above vs just below the threshold. Sharp RD: everyone above threshold is treated. Fuzzy RD: probability of treatment jumps at threshold (some non-compliance). RD gives local average treatment effect (LATE) near the threshold.
- 6Instrumental Variables (IV): an instrument Z is correlated with treatment T but affects outcome Y only through T (exclusion restriction). Two-stage least squares (2SLS): Stage 1: regress T on Z (and controls) to get predicted treatment Tฬ. Stage 2: regress Y on Tฬ. This gives a consistent estimate of the causal effect even with unmeasured confounders. Good instruments are rare and hard to defend โ this is always tested in interviews.
- 7Synthetic control: construct a 'synthetic' version of the treated unit as a weighted combination of control units that best matches the pre-treatment trajectory. Compare the treated unit's post-treatment outcome to the synthetic control. Used for: state/country-level policy evaluation (only one treated unit, can't randomize). Developed by Abadie, Diamond & Hainmueller for California tobacco control evaluation.
- 8Common validity threats: (1) Selection bias โ treated units differ systematically from controls in ways not captured. (2) Attrition bias โ certain types of units drop out of the study (often non-random). (3) Spillover/SUTVA violations โ treatment affects control units. (4) Anticipation effects โ units change behavior before treatment starts, contaminating the pre-period baseline. (5) Parallel trends violation in DiD โ the groups were on diverging trends before treatment.
Key Formulas
| 1 | import numpy as np |
| 2 | import pandas as pd |
| 3 | from scipy import stats |
| 4 | |
| 5 | np.random.seed(42) |
| 6 | N = 1000 # users |
| 7 | |
| 8 | # โโโ Simulate observational study with confounding โโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 9 | # Confounder: user_quality (affects both feature_adoption and retention) |
| 10 | user_quality = np.random.randn(N) |
| 11 | |
| 12 | # Treatment assignment: high-quality users more likely to adopt the feature |
| 13 | treat_prob = 1 / (1 + np.exp(-user_quality)) # logistic function |
| 14 | treatment = np.random.binomial(1, treat_prob) |
| 15 | |
| 16 | # Outcome: retention (affected by treatment AND user_quality) |
| 17 | true_effect = 0.10 # true causal effect of feature: +10pp retention |
| 18 | noise = np.random.randn(N) * 0.15 |
| 19 | retention = 0.60 + 0.15 * user_quality + true_effect * treatment + noise |
| 20 | retention = np.clip(retention, 0, 1) |
| 21 | |
| 22 | df = pd.DataFrame({ |
| 23 | 'treatment': treatment, |
| 24 | 'user_quality': user_quality, |
| 25 | 'retention': retention, |
| 26 | }) |
| 27 | |
| 28 | # โโโ Naive comparison (ignores confounding) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 29 | naive_ate = df[df.treatment==1].retention.mean() - df[df.treatment==0].retention.mean() |
| 30 | print(f"Naive ATE (biased): {naive_ate:.4f} (true = {true_effect:.4f})") |
| 31 | |
| 32 | # โโโ IPW: Inverse Probability Weighting โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 33 | from sklearn.linear_model import LogisticRegression |
| 34 | |
| 35 | ps_model = LogisticRegression() |
| 36 | ps_model.fit(df[['user_quality']], df['treatment']) |
| 37 | propensity = ps_model.predict_proba(df[['user_quality']])[:, 1] |
| 38 | propensity = np.clip(propensity, 0.01, 0.99) # trim extreme weights |
| 39 | |
| 40 | df['weight'] = np.where(df.treatment == 1, |
| 41 | 1 / propensity, |
| 42 | 1 / (1 - propensity)) |
| 43 | |
| 44 | ipw_ate = ((df.treatment * df.retention * df.weight).sum() / df[df.treatment==1].weight.sum() - |
| 45 | ((1-df.treatment) * df.retention * df.weight).sum() / df[df.treatment==0].weight.sum()) |
| 46 | print(f"IPW ATE (debiased): {ipw_ate:.4f} (true = {true_effect:.4f})") |
| 47 | |
| 48 | # โโโ Difference-in-Differences โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 49 | # Simulate panel data: N users, 2 periods (before/after feature launch) |
| 50 | # Treatment group: received feature. Control: never received. |
| 51 | T_treat = np.where(treatment == 1, 0.70, 0.55) # post-treatment outcome |
| 52 | T_ctrl = np.where(treatment == 1, 0.60, 0.55) # pre-treatment outcome (same trend for both) |
| 53 | |
| 54 | # Treatment group gets +0.10 post, control gets +0.02 (secular trend) |
| 55 | post_treat = T_treat + 0.10 * treatment + 0.02 + np.random.randn(N)*0.05 |
| 56 | pre_treat = T_ctrl + np.random.randn(N) * 0.05 |
| 57 | |
| 58 | did_estimate = ((post_treat[treatment==1].mean() - pre_treat[treatment==1].mean()) - |
| 59 | (post_treat[treatment==0].mean() - pre_treat[treatment==0].mean())) |
| 60 | |
| 61 | print(f"DiD estimate: {did_estimate:.4f} (true = 0.10)") |
| 62 | |
| 63 | # โโโ Test parallel trends assumption โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 64 | # Parallel trends: before treatment, both groups should have same trend |
| 65 | # Visual check: plot pre-period trends for both groups |
| 66 | print("\nParallel trends check (pre-period):") |
| 67 | print(f" Treatment pre-mean: {pre_treat[treatment==1].mean():.4f}") |
| 68 | print(f" Control pre-mean: {pre_treat[treatment==0].mean():.4f}") |
| 69 | # If these differ significantly, parallel trends may be violated |
Implements two key causal inference methods: (1) IPW โ logistic regression to estimate propensity scores, then inverse-probability weighting to create a pseudo-population where treatment is unconfounded. The naive estimate is biased by ~50%; IPW recovers the true 10% effect. (2) DiD โ subtracts the control group's time trend from the treatment group's change, isolating the treatment effect from secular trends.
Worked Interview Problems
3 problemsProblem
A feature was shipped to all premium users last year. You want to know if it caused higher retention. You can't retroactively randomize. What's your approach?
Solution
First, understand why you can't randomize and what variation exists. Premium users chose to be premium โ they differ from free users in many ways (confounding). There's no clean natural experiment here.
Propensity score matching or IPW: model P(premium | user_characteristics) using pre-feature-launch data. Weight or match users to create a pseudo-balanced comparison. This addresses observed confounders but NOT unobserved ones (selection bias).
DiD if you have pre/post data: compare retention trends for premium users (who got the feature) vs free users (who didn't) before and after the launch. Requires parallel trends assumption โ test by plotting both groups' trends in the pre-period.
Regression discontinuity if there's a threshold: if users became premium by passing a spending threshold (e.g., $50 total spend), compare users just above and below the threshold. Treats the threshold as a quasi-random assignment.
Honest communication: state clearly what each method identifies and its assumptions. 'Our DiD estimate assumes that in the absence of the feature, premium and free user retention would have trended similarly โ we validated this in the 6 months prior to launch.' Acknowledge the limits โ causal claims from observational data are always weaker than from experiments.
Answer
Approach depends on data: (1) Propensity score methods for cross-sectional comparison (removes observed confounders). (2) DiD if pre-period data exists (controls for time-invariant group differences). (3) RD if there's a threshold. Be explicit about assumptions and limitations of each.
Common Mistakes
Confusing correlation with causation and not checking for confounders. 'Users who use feature X have 30% higher retention' โ but feature X might be adopted by power users who would have high retention anyway. Always ask: is there a confounder?
Using DiD without validating the parallel trends assumption. Parallel trends must be checked empirically by plotting pre-period trends for both groups. Without this check, DiD estimates are meaningless.
Misidentifying the causal estimand. ATE (average effect for all) vs ATT (average effect for those who were treated) vs LATE (local average effect near a threshold). IV estimates LATE, DiD estimates ATT if trends are parallel. Confusing these leads to wrong interpretations.
Trimming propensity scores too aggressively or too loosely. Extreme propensity scores (near 0 or 1) indicate users with almost no comparators โ including them creates high-variance, unreliable estimates. Trim at [0.01, 0.99] and check the common support overlap visually.
Not accounting for spillover effects in network settings. If treated users can share the feature's effects with control users (e.g., through social interaction), the measured ATT will underestimate the true treatment effect because controls are partially treated.