Core Theory

• 1Bayes' theorem and the posterior: P(θ|D) = P(D|θ) P(θ) / P(D). Posterior ∝ Likelihood × Prior. P(θ) is the prior — your belief about θ before seeing data. P(D|θ) is the likelihood — the probability of observing data D given parameter θ. P(θ|D) is the posterior — your updated belief after seeing data. P(D) = ∫ P(D|θ) P(θ) dθ is the marginal likelihood (evidence) — a normalizing constant that makes the posterior a valid distribution. The key Bayesian insight: parameters are random variables with distributions, not fixed unknown constants (the frequentist view). This lets you make direct probability statements: 'P(treatment is better than control | data) = 0.87.'
• 2Conjugate priors: When the prior and posterior belong to the same distributional family, we say the prior is conjugate to the likelihood — conjugacy gives closed-form posteriors, avoiding numerical integration. Beta-Binomial: If X ~ Binomial(n, p) and prior p ~ Beta(α, β), then posterior p|X ~ Beta(α + X, β + n - X). Perfect for conversion rates, CTR. Gamma-Poisson: If X ~ Poisson(λ) and prior λ ~ Gamma(α, β), then posterior λ|X ~ Gamma(α + ΣX, β + n). Perfect for event counts, arrival rates. Normal-Normal: If X ~ N(μ, σ²) with known σ² and prior μ ~ N(μ₀, τ²), then posterior μ|X ~ N(posterior_mean, posterior_variance) with known closed form. Knowing which prior conjugates to which likelihood is a common interview question.
• 3Credible intervals vs confidence intervals: A 95% Bayesian credible interval [a, b] means: given the data, there is a 95% probability that the true parameter lies in [a, b]. This is the natural interpretation people mistakenly assign to frequentist CIs. A 95% frequentist confidence interval means: if you repeated the experiment many times and constructed a CI each time, 95% of those intervals would contain the true value. The specific interval you observe either does or does not contain the true value — no probability statement about this specific interval is valid in the frequentist framework. In Bayesian analysis, the HDI (Highest Density Interval) is the shortest interval containing 95% of the posterior mass — preferred over equal-tail intervals when the posterior is skewed.
• 4MCMC — Metropolis-Hastings: Used when the posterior has no closed form. Algorithm: (1) Start at θ₀. (2) Propose θ* from a proposal distribution q(θ*|θ_t). (3) Compute acceptance ratio r = [P(D|θ*) P(θ*)] / [P(D|θ_t) P(θ_t)] × [q(θ_t|θ*) / q(θ*|θ_t)]. (4) Accept θ* with probability min(1, r); otherwise stay at θ_t. (5) Repeat. Under mild conditions, the chain converges to the target posterior distribution. Convergence diagnostics: R-hat (Gelman-Rubin statistic) < 1.1 indicates convergence across multiple chains. Effective sample size (ESS) should be > 400 for reliable posterior estimates. Trace plots should look like 'fuzzy caterpillars' — mixing well without getting stuck.
• 5Gibbs sampling: A special case of MCMC where, for a model with multiple parameters θ = (θ₁, θ₂, ..., θ_k), you iteratively sample each parameter from its full conditional distribution P(θ_i | θ_{-i}, D) while holding all other parameters fixed. Requires knowing the full conditionals in closed form. More efficient than Metropolis-Hastings when conditionals are tractable. Used in Bayesian network inference and LDA (Latent Dirichlet Allocation). Hamiltonian Monte Carlo (HMC) uses gradient information to make longer, more efficient proposals — the basis of Stan and PyMC's NUTS sampler, which is now the production standard for Bayesian modeling.
• 6Variational inference (VI): An alternative to MCMC that frames posterior inference as optimization. Approximate the true posterior P(θ|D) with a simpler distribution q(θ; φ) from a variational family (e.g., mean-field: q(θ) = Πᵢ q(θᵢ)). Minimize the KL divergence KL(q || P) by maximizing the Evidence Lower BOund (ELBO): ELBO = E_q[log P(D|θ)] - KL(q(θ) || P(θ)). Maximizing the ELBO is equivalent to minimizing KL(q || P). VI is faster than MCMC (optimization vs sampling) but approximate — tends to underestimate posterior variance. Scalable VI (stochastic VI, ADVI) enables Bayesian inference on large datasets. Used in production at Uber, Netflix for large-scale latent variable models.
• 7Bayesian A/B testing — sequential decision making: The core advantage over frequentist testing: you can stop the experiment early at any time without inflating Type I error, because you're computing P(treatment > control | data) directly from the posterior, not a p-value. Setup: model control conversion rate p_A ~ Beta(α₀, β₀), treatment p_B ~ Beta(α₀, β₀) with weak prior. After observing n_A conversions in N_A trials: p_A|data ~ Beta(α₀ + n_A, β₀ + N_A - n_A). P(p_B > p_A | data) = ∫∫ 1[p_B > p_A] P(p_A|data) P(p_B|data) dp_A dp_B — computable analytically for Beta distributions or via Monte Carlo sampling. Expected loss: E[loss of choosing A | data] = E[max(p_B - p_A, 0) | data]. Decision rule: ship treatment when P(p_B > p_A | data) > 0.95 or when expected loss < ε.
• 8Hierarchical models (partial pooling): Suppose you're modeling conversion rates for 50 different product categories. Complete pooling (estimate one global rate) ignores between-category variation. No pooling (estimate each category independently) ignores information shared across categories — small categories get noisy estimates. Hierarchical model: model each category's rate θ_k ~ Beta(α, β) where (α, β) are drawn from a hyperprior — categories share information through the hyperprior. The result is partial pooling: small categories are shrunk toward the global mean, large categories retain their own estimate. This is empirical Bayes in a full Bayesian formulation. Directly tested at companies running experiments across many product verticals (Airbnb) or regions (Uber).
• 9Thompson sampling for multi-armed bandits: The exploration-exploitation dilemma: should you exploit the arm with the best observed reward, or explore other arms that might be better? Thompson sampling: at each round, sample θ_k from the posterior P(θ_k | observed rewards for arm k). Pull the arm with the highest sampled θ_k. The uncertainty in the posterior naturally drives exploration (arms with uncertain posteriors are sometimes sampled high) while the posterior mean drives exploitation. For a Bernoulli bandit with Beta-Binomial conjugacy: initialize p_k ~ Beta(1,1). After observing s_k successes and f_k failures for arm k: sample θ_k ~ Beta(1+s_k, 1+f_k) each round. Thompson sampling is asymptotically optimal (achieves Lai-Robbins lower bound on cumulative regret). Used at Netflix for A/B test allocation, at Spotify for playlist ranking.
• 10Gaussian Processes (GP): A GP is a distribution over functions. Any finite collection of function values f(x₁), ..., f(xₙ) follows a multivariate Gaussian. Specified by: mean function m(x) and covariance (kernel) function k(x, x'). Common kernels: RBF (squared exponential) k(x,x') = σ² exp(-||x-x'||²/(2l²)) — smooth functions; Matérn — controls smoothness via ν parameter; periodic kernel for periodic patterns. GP regression: given observations y = f(X) + ε (ε ~ N(0, σ²_noise)), the posterior predictive for new point x* is also Gaussian with closed-form mean and variance. GP is the non-parametric Bayesian alternative to kernel regression — it quantifies uncertainty in predictions, not just point estimates. Limitation: O(n³) computation for n training points — approximate methods (sparse GPs, inducing points) scale to larger datasets.
• 11Bayesian optimization: Uses a GP as a surrogate model to optimize expensive black-box functions (e.g., hyperparameter tuning for ML models). Algorithm: (1) Build a GP posterior over the objective function using evaluated points. (2) Maximize an acquisition function that balances exploration and exploitation to select the next point to evaluate. (3) Evaluate the expensive function at that point. (4) Update the GP. (5) Repeat. Acquisition functions: Expected Improvement (EI) = E[max(f(x) - f_best, 0)]; Upper Confidence Bound (UCB) = μ(x) + κ σ(x); Probability of Improvement (PI). Bayesian optimization typically finds good hyperparameters in 50-100 function evaluations vs 1000s for random search. Used in AutoML systems (Google Vizier, Optuna, Spearmint).
• 12MAP estimation vs MLE: Maximum Likelihood Estimation (MLE) finds θ_MLE = argmax_θ P(D|θ). Maximum A Posteriori (MAP) finds θ_MAP = argmax_θ P(θ|D) = argmax_θ P(D|θ)P(θ). MAP is equivalent to MLE with a regularization term equal to -log P(θ). Connection: L2 regularization (Ridge) = MAP with Gaussian prior. L1 regularization (LASSO) = MAP with Laplace prior. MAP gives a point estimate (not a full posterior) — it does not quantify uncertainty. As n → ∞, MAP and MLE converge because the likelihood dominates the prior. MAP is a computationally tractable Bayesian approximation when full posterior inference is too expensive.