Core Theory

• 1Missing data taxonomy (MCAR/MAR/MNAR): Missing Completely At Random (MCAR) means missingness is independent of both observed and unobserved data — a sensor randomly drops packets. Safe to drop rows or impute with mean/median. Missing At Random (MAR) means missingness depends on observed data but not on the missing value itself — older users less likely to fill in income, but age is recorded. Imputation using other observed features (KNN, MICE) is valid. Missing Not At Random (MNAR) means missingness depends on the missing value itself — high-earners skip the income field. Simple imputation introduces bias; must model the missingness mechanism itself or use domain-specific rules. Always test: does the missingness pattern correlate with the target? If yes, add a binary indicator feature for missingness.
• 2Imputation strategies: Mean/median imputation is fast but reduces variance and distorts correlations — never use for MNAR. KNN imputation finds the K nearest complete observations and averages their values; works well for MAR with moderate missingness (<30%), expensive at O(n²). MICE (Multiple Imputation by Chained Equations) runs m separate imputed datasets, trains models on each, and pools results — the gold standard for MAR. Indicator variables for missingness: always add a binary column `feature_is_missing` alongside imputed values so the model can learn that missingness itself is informative. For MNAR tree-based models, the 'missing' category itself often becomes a strong predictor; XGBoost/LightGBM handle NaN natively by learning the optimal direction for missing values.
• 3Categorical encoding: One-hot encoding creates k binary columns for k categories — fine for low-cardinality (<20 levels) nominals, but explodes with high-cardinality features. Target encoding replaces each category with the mean target value (for regression: mean(y | category); for classification: P(y=1 | category)). Without regularization, target encoding leaks the target — small categories will overfit perfectly. Regularization: (1) add Gaussian noise to the category mean, (2) blend with global mean via n_smoothing: encoded_value = (n × category_mean + k × global_mean) / (n + k), (3) leave-one-out encoding (exclude the current row when computing the category mean). Ordinal encoding preserves order (cold < warm < hot) and is appropriate for ordinal features. Entity embeddings (as in the Rossmann competition) learn dense low-dimensional representations of categories via a neural network embedding layer — outperforms one-hot for very high-cardinality features (user IDs, product IDs).
• 4Feature scaling: StandardScaler (z-score normalization: x' = (x - μ) / σ) centers features at mean 0 with unit variance — required for distance-based algorithms (KNN, SVM, k-means), gradient descent-based models (linear/logistic regression, neural nets), and PCA. MinMaxScaler (x' = (x - min) / (max - min)) maps to [0,1] — useful when you need bounded outputs, but very sensitive to outliers (one outlier at x=1000 compresses all other values). RobustScaler uses median and IQR (x' = (x - median) / IQR) — best choice when outliers are present. Tree-based models (decision trees, random forests, gradient boosting) are completely scale-invariant — scaling provides no benefit. Never fit the scaler on the full dataset before the train/test split; fit only on train, transform both train and test.
• 5Feature interactions and polynomial features: Linear models cannot capture interactions between features — if CTR depends on (ad_format × user_segment), a linear model treating them separately will miss this. Polynomial features: PolynomialFeatures(degree=2) adds all products x_i × x_j and squares x_i² — for p features, generates O(p²) interaction terms. Impractical beyond p~100 features. Manual interactions: domain-driven multiplication of semantically meaningful pairs (price_per_sqft = price / sqft). For tree-based models, interactions are captured automatically via tree splits — polynomial features add no benefit. For neural networks, feature interactions are learned implicitly in deeper layers.
• 6Date/time features: Raw timestamps are useless to most models. Extract: hour_of_day (0-23), day_of_week (0-6), month (1-12), is_weekend (bool), is_holiday (bool), days_since_last_purchase, days_until_event. Cyclical encoding for periodic features: sin/cos encoding prevents the model from treating hour 23 and hour 0 as far apart. hour_sin = sin(2π × hour / 24), hour_cos = cos(2π × hour / 24). Lag features for time series: lag_1 = value at t-1, lag_7 = value at t-7 (same day last week), rolling_mean_7d = 7-day moving average. Be careful: lag features for time series require special cross-validation (time-based splits, not random splits) to avoid leakage.
• 7Data leakage — the #1 source of production model failures: Target leakage occurs when features are derived from or correlated with the target in a way that wouldn't be available at prediction time. Example: 'days_in_hospital' as a feature for predicting hospital readmission — you don't know this when the patient is admitted. Train-test contamination occurs when preprocessing is fit on the full dataset before splitting: if you compute the global mean for imputation on all data, the test set's information leaks into training statistics. Prevention: wrap all preprocessing in a scikit-learn Pipeline, fit Pipeline.fit() only on training data. Time-based contamination: using future data to compute features for the past (e.g., rolling average that looks forward in time). Feature store contamination: if the feature store serves stale features or uses different computation logic for training vs serving, predictions will be systematically wrong (training-serving skew).
• 8Distribution shift detection and handling: Covariate shift occurs when P(X) changes but P(y|X) stays the same — input distribution drifts, but the relationship between features and target remains stable. Example: user age distribution shifts as the app grows. Detection: monitor feature statistics (mean, std, percentiles) over time; compute KL divergence or Population Stability Index (PSI > 0.2 indicates significant shift). PSI = Σ (Actual% - Expected%) × ln(Actual%/Expected%). Concept drift occurs when P(y|X) changes — the relationship between features and target changes. Example: fraud patterns evolve as fraudsters adapt. Detection: monitor model performance (AUC, accuracy) on labeled data with short lags. Label shift occurs when P(y) changes — class priors shift. Handling: retrain on recent data, apply importance weighting (weight training samples by P_new(X) / P_old(X)), use concept drift detectors (ADWIN, DDM, KSWIN).
• 9Feature selection methods: Filter methods use statistical tests independent of any model. Mutual Information measures non-linear dependence between feature and target: MI(X;Y) = Σ P(x,y) log[P(x,y)/(P(x)P(y))]. Chi-squared test for categorical features vs categorical target. Pearson correlation for linear dependence. Wrapper methods use model performance to select features. Recursive Feature Elimination (RFE) trains a model, removes the least important feature, and repeats — exact but expensive at O(p) model trainings. Embedded methods perform selection during model training. LASSO (L1 regularization) shrinks unimportant feature coefficients to exactly zero — performs feature selection implicitly. Tree feature importance measures the average gain in purity (Gini or entropy) from splits on each feature — but biased toward high-cardinality features. SHAP-based importance is unbiased and more reliable. Permutation importance: shuffle each feature column and measure drop in validation performance — model-agnostic and unbiased.
• 10Class imbalance handling: Most real-world classification problems are imbalanced (fraud: 0.1%, churn: 5%, click: 1%). Never evaluate on accuracy alone — a model predicting all negatives achieves 99.9% accuracy on 0.1% fraud data. Use AUC-ROC, Precision-Recall AUC, or F1. Oversampling: SMOTE (Synthetic Minority Over-sampling Technique) creates synthetic minority samples by interpolating between real minority samples in feature space — better than simple replication. ADASYN (Adaptive Synthetic Sampling) generates more samples in harder-to-learn regions. Undersampling: random undersampling of the majority class (information loss); Tomek Links removes majority samples adjacent to minority samples (cleans decision boundary). Class weights: set class_weight='balanced' in sklearn or scale_pos_weight in XGBoost — equivalent to reweighting the loss function without modifying the data. Threshold tuning: the default 0.5 decision threshold is wrong for imbalanced classes; find the threshold that maximizes F1 or satisfies your precision/recall requirement on the validation set.
• 11Feature stores in production: A feature store is a centralized repository for computing, storing, and serving ML features consistently. The core problem it solves: training-serving skew — if you compute features differently in the training pipeline (batch, using all historical data) vs the serving pipeline (real-time, per-request), model performance degrades. Components: offline store (Hive, Redshift, BigQuery) for training data retrieval, online store (Redis, DynamoDB, Cassandra) for low-latency serving (<10ms), feature registry (metadata: feature name, owner, computation logic, freshness SLA), feature pipelines (Spark/Flink jobs that compute features from raw events). Point-in-time correctness: when generating training data for a model that will predict at time T, all features must be computed using only data available before time T — no future leakage. Examples: Feast (open-source), Tecton (enterprise), Hopsworks, Vertex AI Feature Store. At Amazon, the ML platform computes ~trillions of feature values per day for ranking models.