ML Modeling & Model Selection

Applied Scientists are the bridge between ML research and business impact. You must be able to translate a business problem into an ML objective, select the right model, evaluate it correctly, and interpret it for stakeholders. 'Build a model to predict X' is the most common Applied Scientist interview format.

ML fundamentalsStatisticsPython/sklearnFeature engineeringXGBoostModel SelectionSHAPCalibrationFeature EngineeringAUC

Core Theory

1Problem framing: before any model: (1) define the ML objective (maximize P(click)? minimize churn risk? rank items by relevance?), (2) define the training label (what is a positive? what is a negative?), (3) identify the prediction horizon (predict next 30 days? next session? real-time?), (4) identify constraints (latency, interpretability requirements, minimum recall threshold). Wrong objective → wrong model → no business impact.
2Model selection framework: start simple. Logistic regression baseline first — it's interpretable, fast, and often surprisingly competitive. Then try GBDT (XGBoost/LightGBM) for structured/tabular data. Neural networks only when: data is unstructured (text, images), or dataset is very large (>1M samples), or feature interactions are extremely complex. Deep learning for tabular data rarely beats well-tuned GBDT.
3XGBoost/LightGBM key hyperparameters: max_depth (3-6 for most tasks — deeper = more overfitting), learning_rate (0.01-0.1, lower = more trees needed), n_estimators (use early stopping, not a fixed number), subsample (0.7-0.9, bagging fraction), colsample_bytree (0.7-0.9, feature subsampling per tree), min_child_weight (regularizes leaf nodes), reg_lambda/reg_alpha (L2/L1 on leaf weights).
4Evaluation for classification: AUC-ROC (threshold-invariant, probability that model ranks a random positive above a random negative), AUC-PR (better for imbalanced data — measures precision-recall tradeoff), F1/F-beta at business threshold, log-loss (measures calibration quality). Calibration: are predicted probabilities accurate? A model can have AUC=0.92 but terrible calibration (predicted 90% probability for events that occur 50% of the time).
5SHAP values: SHapley Additive exPlanations. For each prediction, φ_i = the contribution of feature i. Properties: (1) efficiency (Σφ_i = model output - baseline), (2) consistency (if model relies more on feature i, its SHAP value can only increase), (3) null player (unused features get SHAP=0). Tree SHAP is O(TLD) — fast enough for large models. More reliable than gain-based feature importance, which is biased toward high-cardinality features.
6Distribution shift / train-serve skew: model trained on historical data but deployed on future data with different distributions. Symptoms: high training AUC, low production AUC. Causes: (1) covariate shift — input distribution changed (e.g., new user demographics), (2) label shift — proportion of positives changed (e.g., seasonality), (3) concept drift — relationship between X and Y changed (e.g., user behavior evolved). Detection: compare training vs serving input distributions (PSI), monitor prediction score distribution.
7Imbalanced classes: when positive class is rare (<5%), models can achieve high accuracy by predicting all negatives. Solutions: (1) class weighting (class_weight='balanced' in sklearn, scale_pos_weight in XGBoost), (2) resampling (oversample minority with SMOTE, undersample majority), (3) threshold tuning (move the classification threshold from 0.5 to optimize the business metric), (4) anomaly detection framing (for <1% positive rate). Evaluation: always use AUC-PR or F1 at appropriate threshold, never accuracy.
8Feature engineering for tabular ML: (1) encoding categoricals — one-hot for low cardinality (<20 categories), target encoding for high cardinality (mean of Y per category, with smoothing to prevent leakage), entity embeddings for neural networks. (2) DateTime features — day-of-week, hour-of-day, time since last event, rolling statistics. (3) Interaction features — multiply two features for XGBoost (trees can find interactions but benefit from explicit help). (4) Text features — TF-IDF, sentence embeddings (FinBERT, SBERT) for semantic search.

Key Formulas

01XGBoost objective:

L(\phi) = \sum_i l(y_i, \hat{y}_i) + \sum_k (\gamma T_k + \frac{1}{2}\lambda||w_k||^2)

02SHAP:

g(x') = \phi_0 + \sum_i \phi_i x'_i

\sum \phi_i = f(x) - E[f(X)]

03Log-loss:

L = -\frac{1}{n}\sum_i [y_i \log \hat{p}_i + (1-y_i)\log(1-\hat{p}_i)]

04Target encoding:

\hat{\mu}_k = \frac{n_k \bar{y}_k + m \bar{y}_{global}}{n_k + m}

(smoothed)

05F-beta:

F_\beta = (1+\beta^2) \cdot \frac{precision \cdot recall}{\beta^2 \cdot precision + recall}

python

1	import numpy as np
2	import pandas as pd
3	from sklearn.datasets import make_classification
4	from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
5	from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss
6	from sklearn.calibration import CalibratedClassifierCV
7	from sklearn.linear_model import LogisticRegression
8
9	np.random.seed(42)
10
11	# ─── Create imbalanced churn prediction dataset ───────────────────────────────
12	X, y = make_classification(n_samples=50_000, n_features=20, n_informative=8,
13	weights=[0.92, 0.08], random_state=42)
14	X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
15	stratify=y, random_state=42)
16
17	# ─── Baseline: Logistic Regression ────────────────────────────────────────────
18	lr = LogisticRegression(class_weight='balanced', max_iter=500)
19	lr.fit(X_tr, y_tr)
20	lr_proba = lr.predict_proba(X_te)[:, 1]
21
22	# ─── Main model: XGBoost with class imbalance handling ────────────────────────
23	try:
24	from xgboost import XGBClassifier
25	pos_weight = (y_tr == 0).sum() / (y_tr == 1).sum() # ~11.5 for 8% churn
26	xgb = XGBClassifier(
27	n_estimators=300, max_depth=4, learning_rate=0.05,
28	subsample=0.8, colsample_bytree=0.8,
29	scale_pos_weight=pos_weight, # handle class imbalance
30	eval_metric='logloss', random_state=42, verbosity=0
31	)
32	xgb.fit(X_tr, y_tr, eval_set=[(X_te, y_te)], verbose=False)
33	xgb_proba = xgb.predict_proba(X_te)[:, 1]
34	model_name = "XGBoost"
35	proba = xgb_proba
36	except ImportError:
37	from sklearn.ensemble import GradientBoostingClassifier
38	model = GradientBoostingClassifier(n_estimators=100, max_depth=3, random_state=42)
39	model.fit(X_tr, y_tr)
40	proba = model.predict_proba(X_te)[:, 1]
41	model_name = "GBM (sklearn)"
42
43	# ─── Evaluation ───────────────────────────────────────────────────────────────
44	print(f"\n{'Metric':<25} \| {'Logistic':>10} \| {model_name:>12}")
45	print("-" * 52)
46	print(f"{'AUC-ROC':<25} \| {roc_auc_score(y_te, lr_proba):>10.4f} \| {roc_auc_score(y_te, proba):>12.4f}")
47	print(f"{'AUC-PR':<25} \| {average_precision_score(y_te, lr_proba):>10.4f} \| {average_precision_score(y_te, proba):>12.4f}")
48	print(f"{'Brier score':<25} \| {brier_score_loss(y_te, lr_proba):>10.4f} \| {brier_score_loss(y_te, proba):>12.4f}")
49
50	# ─── Threshold tuning for business objective ──────────────────────────────────
51	from sklearn.metrics import precision_recall_curve, f1_score
52
53	precisions, recalls, thresholds = precision_recall_curve(y_te, proba)
54
55	# Business scenario: calling churners costs $2/call, saving a churner is worth $50
56	# Expected value = recall * $50 * base_rate - precision_mistake_rate * $2
57	# Simplified: find threshold maximizing F2 (recall-weighted)
58	f2_scores = [(1+4) * p * r / (4*p + r + 1e-8) for p, r in zip(precisions, recalls)]
59	best_thresh_idx = np.argmax(f2_scores)
60	best_threshold = thresholds[best_thresh_idx]
61
62	y_pred = (proba >= best_threshold).astype(int)
63	print(f"\nOptimal threshold (F2): {best_threshold:.3f}")
64	print(f" Precision: {precisions[best_thresh_idx]:.3f}")
65	print(f" Recall: {recalls[best_thresh_idx]:.3f}")
66	print(f" Users flagged: {y_pred.sum()} / {len(y_pred)} ({y_pred.mean()*100:.1f}%)")

Complete churn prediction pipeline. Key design choices: (1) scale_pos_weight for class imbalance — tells XGBoost to weight positive (churner) examples ~11.5× higher; (2) AUC-PR instead of AUC-ROC for the imbalanced 8% churn rate; (3) threshold tuning with F2 score (β=2, recall-weighted) to match a business scenario where missing a churner is more costly than false alarms; (4) Brier score to measure probability calibration quality.

Worked Interview Problems

3 problems

Problem

You're an Applied Scientist at a subscription product. Leadership wants to proactively identify users likely to cancel in the next 30 days to offer them a retention incentive. Walk through your complete approach.

Solution

Problem framing: label = 1 if user cancels within 30 days of today, 0 otherwise. Prediction horizon: 30 days. Positive class rate: ~5% monthly churn. Constraint: the retention team can contact 2,000 users/week (precision matters, not just recall).

Feature engineering: (1) Engagement features — login frequency (7/14/30 day), feature usage rate, content consumed, time since last session. (2) Account features — subscription tier, tenure, payment method, past support contacts. (3) Trend features — is engagement increasing or decreasing? ratio of (last 7d logins / prior 7d logins). (4) NLP features — sentiment of last support ticket (if any).

Model selection: logistic regression baseline (interpretable, helps understand feature importance direction). XGBoost main model (handles non-linear interactions, robust to skewed distributions). Train/val/test split: last 3 months as test (temporal split — never shuffle for time-based labels), previous 12 months for training, 1 month validation.

Evaluation: AUC-PR (imbalanced), precision at recall=30% (what fraction of contacted users actually churn?), lift chart (are we targeting the right people?). Business metric: expected ROI = (precision × 2000 × $50 retention value) - (2000 ×$ 2 contact cost). Threshold: set based on this ROI calculation.

SHAP analysis: compute SHAP values to understand model drivers. Report to PM: 'The top predictors are declining login frequency (-0.08 SHAP), no mobile app use in 14 days (-0.06), and 2+ support contacts in 30 days (+0.05 toward churn).' This makes the model actionable — the retention team knows what to address.

Answer

Full pipeline: frame (30-day binary label, 5% churn rate, 2k contact limit) → feature engineering (engagement trends, account health, NLP) → temporal train/test split → XGBoost with scale_pos_weight → evaluate on AUC-PR + precision at recall threshold → tune threshold for business ROI → SHAP for actionable interpretation.

Common Mistakes

Using accuracy as the evaluation metric for imbalanced datasets. With 92% negatives, a model that predicts all negative gets 92% accuracy. AUC-ROC, AUC-PR, or F1 at a meaningful threshold are the right metrics.

Not doing a temporal train/test split for time-series-like labels (e.g., 30-day churn). If you randomly shuffle and split, future information leaks into training data. Always train on earlier data and test on later data.

Reporting AUC-ROC for imbalanced data as if it's meaningful. For 1% positive rate, a random model gets AUC=0.50 but AUC-PR=0.01 (= base rate). AUC-ROC can look fine (0.85) even when precision at any practical threshold is terrible. Use AUC-PR for imbalanced problems.

Using target encoding without a holdout or smoothing — target encoding (mean outcome per category) leaks label information if computed on the training set and used in the same model. Use out-of-fold target encoding: for fold k, compute encoding using all other folds.

Treating feature importance from a trained model as a causal explanation. Feature importance says 'the model uses this feature to predict.' It doesn't mean 'this feature causes the outcome.' Always distinguish predictive from causal claims.

Back to ML Hub

Causal Inference SQL & Data Analysis