ML Engineer
ML Fundamentals
ML fundamentals are tested in every phone screen and first round. You must be able to explain bias-variance tradeoff precisely, derive regularization effects, choose the right evaluation metric, and compare tree-based models โ all without looking anything up.
Core Theory
- 1Bias-variance decomposition: MSE = Biasยฒ + Variance + Irreducible noise. Bias: error from wrong assumptions (underfitting). Variance: error from sensitivity to training data fluctuations (overfitting). High-bias models: linear regression on nonlinear data. High-variance models: deep decision trees. The tradeoff: reducing bias often increases variance and vice versa. Ensemble methods (bagging, boosting) reduce variance and bias respectively.
- 2L1 (Lasso) regularization: adds ฮปฮฃ|wแตข| to loss. Gradient: ฮปยทsign(wแตข). The sign function is discontinuous at 0, which creates sparsity โ some weights are driven exactly to 0 (feature selection). Useful when you believe most features are irrelevant. L2 (Ridge): adds ฮปฮฃwแตขยฒ to loss. Gradient: 2ฮปwแตข. Shrinks all weights proportionally โ never zero unless forced. Better when all features contribute. ElasticNet: L1 + L2 combined.
- 3Cross-validation: k-fold โ split data into k folds, train on k-1 and test on 1, rotate. Average metrics across folds. Stratified k-fold preserves class ratio in each fold โ essential for imbalanced data. Leave-one-out (LOO): k = n, computationally expensive but low variance. Time-series CV: always train before test, never shuffle โ use TimeSeriesSplit or PurgedKFold.
- 4Loss functions by task: regression โ MSE (penalizes large errors quadratically), MAE (robust to outliers, non-differentiable at 0), Huber (MSE for small errors, MAE for large โ best of both). Classification โ cross-entropy (proper scoring rule, gradient = ลท - y), hinge loss (SVM, creates margin). Ranking โ pairwise (RankNet), listwise (LambdaMART, NDCG-optimized).
- 5Evaluation metrics: classification โ accuracy (misleading for imbalanced data), precision (TP/(TP+FP)), recall (TP/(TP+FN)), F1 (harmonic mean of precision/recall), AUC-ROC (threshold-invariant, probability-based), AUC-PR (better for highly imbalanced data). Calibration (Brier score): are predicted probabilities accurate? A model with AUC=0.95 can still be badly calibrated.
- 6Decision trees: split by information gain (ID3, C4.5) or Gini impurity (CART). Information gain = H(parent) - ฮฃ wแตขH(child_i). Gini = 1 - ฮฃ pแตขยฒ. Prone to overfitting (zero training error with unlimited depth). Regularization: max_depth, min_samples_split, min_samples_leaf, max_features.
- 7Random Forest vs Gradient Boosting: Random Forest = bagging (parallel training of deep trees on bootstrap samples with random feature subsets). Reduces variance. GBM = boosting (sequential: each tree fits the residuals of all previous trees). Reduces bias. XGBoost/LightGBM add regularization to GBM objective. Rule: Random Forest for noisy data, GBM for clean structured data.
- 8Gradient descent variants: batch GD (full dataset per step โ exact gradient, slow), stochastic GD (one sample โ noisy gradient, fast, escapes local minima), mini-batch GD (best of both). Momentum: accumulate gradient history to dampen oscillations. Adam: adaptive learning rates per parameter via first and second moment estimates.
Key Formulas
| 1 | import numpy as np |
| 2 | from sklearn.tree import DecisionTreeClassifier |
| 3 | from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier |
| 4 | from sklearn.linear_model import LogisticRegression |
| 5 | from sklearn.datasets import make_classification |
| 6 | from sklearn.model_selection import cross_val_score, StratifiedKFold |
| 7 | from sklearn.metrics import precision_recall_curve, average_precision_score, roc_auc_score |
| 8 | |
| 9 | np.random.seed(42) |
| 10 | |
| 11 | # โโโ Bias-Variance tradeoff across model complexity โโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 12 | X, y = make_classification(n_samples=1000, n_features=20, n_informative=5, |
| 13 | n_redundant=5, random_state=42) |
| 14 | |
| 15 | models = { |
| 16 | "Logistic (high bias)": LogisticRegression(max_iter=500), |
| 17 | "Tree depth=3 (balanced)": DecisionTreeClassifier(max_depth=3), |
| 18 | "Tree depth=20 (high var)": DecisionTreeClassifier(max_depth=20), |
| 19 | "Random Forest (low var)": RandomForestClassifier(n_estimators=100), |
| 20 | "GBM (low bias)": GradientBoostingClassifier(n_estimators=100), |
| 21 | } |
| 22 | |
| 23 | cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) |
| 24 | |
| 25 | print(f"{'Model':<35} | {'Mean AUC':>9} | {'Std AUC':>8}") |
| 26 | print("-" * 60) |
| 27 | for name, model in models.items(): |
| 28 | scores = cross_val_score(model, X, y, cv=cv, scoring='roc_auc') |
| 29 | print(f"{name:<35} | {scores.mean():>9.4f} | {scores.std():>8.4f}") |
| 30 | |
| 31 | # โโโ Imbalanced classification: PR curve vs ROC โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 32 | # Create imbalanced dataset (1% positive class โ fraud-like) |
| 33 | X_imb, y_imb = make_classification(n_samples=10_000, n_features=20, |
| 34 | n_informative=5, weights=[0.99, 0.01], |
| 35 | random_state=42) |
| 36 | |
| 37 | from sklearn.model_selection import train_test_split |
| 38 | X_tr, X_te, y_tr, y_te = train_test_split(X_imb, y_imb, test_size=0.3, |
| 39 | stratify=y_imb, random_state=42) |
| 40 | |
| 41 | clf = GradientBoostingClassifier(n_estimators=100).fit(X_tr, y_tr) |
| 42 | proba = clf.predict_proba(X_te)[:, 1] |
| 43 | |
| 44 | auc_roc = roc_auc_score(y_te, proba) |
| 45 | avg_prec = average_precision_score(y_te, proba) # area under PR curve |
| 46 | |
| 47 | print(f"\nImbalanced dataset (1% positive):") |
| 48 | print(f" AUC-ROC: {auc_roc:.4f} (inflated by large TN count)") |
| 49 | print(f" AUC-PR: {avg_prec:.4f} (better reflects performance on minority class)") |
| 50 | print(f" Baseline PR-AUC (random): {y_te.mean():.4f}") |
| 51 | |
| 52 | # โโโ When to use each metric โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 53 | print("\nMetric selection guide:") |
| 54 | print(" AUC-ROC: use when classes are balanced AND FP/FN costs are symmetric") |
| 55 | print(" AUC-PR: use when positive class is rare (fraud, disease, rare event)") |
| 56 | print(" F1: use when you need a single threshold-dependent metric") |
| 57 | print(" Accuracy: ONLY use when classes are balanced AND costs are equal") |
Demonstrates the bias-variance tradeoff across model families: high-variance deep tree vs. regularized ensemble. The imbalanced classification section shows why AUC-PR is more informative than AUC-ROC when the positive class is rare (1% fraud rate). A random model gets AUC-ROC=0.5 but AUC-PR=0.01 (= base rate) โ the gap shows how misleading ROC is for imbalanced data.
Worked Interview Problems
3 problemsProblem
You have a fraud detection model. 99% of transactions are legitimate. Your model predicts 'not fraud' for every transaction. What is the accuracy? Why is this a problem?
Solution
Accuracy = (TP + TN) / Total. If model predicts 'not fraud' for all 10,000 transactions, and 9,900 are legitimate: Accuracy = 9,900/10,000 = 99%.
But Recall (fraud detection rate) = TP / (TP + FN) = 0 / 100 = 0%. The model detects zero fraudulent transactions. It is completely useless for its actual purpose.
Precision doesn't apply here (no positive predictions). F1 = 2 ร (0 ร undefined) / (0 + undefined) = 0. The model has zero utility.
Better metrics: Precision@k (what fraction of flagged transactions are actually fraud), Recall@k (what fraction of all fraud is caught), AUC-PR (overall tradeoff across thresholds), F1 at business-appropriate threshold.
Practical note: the business determines the threshold based on cost tradeoff: cost_false_negative (missed fraud, bank absorbs loss) vs cost_false_positive (investigating a legitimate transaction, customer inconvenience). This is not a ML decision โ it's a product/business decision.
Answer
Accuracy = 99% but Recall = 0% โ the model is useless. Accuracy is a bad metric when classes are imbalanced. Use AUC-PR or F1 at a business-appropriate threshold for imbalanced classification.
Common Mistakes
Using accuracy for imbalanced classification. If 99% of data is class A and 1% is class B, a model predicting only class A gets 99% accuracy but is useless. Always use AUC-ROC, AUC-PR, or F1 for imbalanced data.
Not using stratified k-fold for imbalanced data. Regular k-fold may put all positive examples in one fold, giving 0% positive class in training and 100% in test โ completely invalid evaluation.
Applying the same train/test split for hyperparameter tuning and final evaluation. Tune hyperparameters on a validation set, report final performance on a held-out test set that was never used during development.
Forgetting to normalize/standardize features before L2 regularization or gradient-based optimization. L2 regularization penalizes all weights equally โ if feature X has range [0,1] and feature Y has range [0,1000], their coefficients will have very different natural magnitudes, and regularization will penalize them unevenly.
Interpreting feature importance from a trained model as causal. Feature importance tells you what the model uses to predict, not what causes the outcome. A model that heavily uses 'employee ID' is picking up on a data leakage artifact, not a causal relationship.