ML Engineer
MLOps & Production Infrastructure
MLOps is the practice of deploying, monitoring, and maintaining ML models in production reliably. Senior MLE roles at every company require fluency with containerization, experiment tracking, model serving, and automated retraining pipelines. The goal: models that are reproducible, observable, and self-healing.
Core Theory
- 1Containerization: Docker packages code + dependencies into a portable image. For ML: pin Python version, pip dependencies (requirements.txt with exact versions), CUDA version, and model artifacts in a single image. Multi-stage builds reduce image size: build stage installs dependencies, final stage copies only what's needed. Kubernetes orchestrates containers at scale — manages replicas, rolling deployments, resource limits (GPU requests).
- 2Experiment tracking: MLflow (or W&B) logs parameters, metrics, and artifacts for each training run. Enables reproducibility (rerun any experiment), comparison (which hyperparameters gave the best val loss?), and audit trails. Model Registry allows staging models through development → staging → production with version control and approval gates.
- 3Model serving architectures: REST API (FastAPI/Flask — simple, good for low QPS), TorchServe (PyTorch models with batching, A/B splitting built-in), NVIDIA Triton (GPU batching, concurrent model execution, multiple frameworks, production standard for high-throughput inference). For LLMs: vLLM (PagedAttention, continuous batching, 2-10× throughput improvement over naive serving).
- 4CI/CD for ML: automated pipeline triggered by code push or new data. Steps: (1) data validation (Great Expectations schema check), (2) unit tests for data pipeline + model code, (3) model training on CI machine, (4) model validation (offline metrics above threshold), (5) staging deployment + integration tests, (6) production canary deployment (5% traffic) → full rollout. Gate at every step.
- 5Feature stores (Feast, Tecton): offline store (Hive/S3 Parquet for training) + online store (Redis for <5ms serving). Point-in-time joins for training data prevent look-ahead bias. Feature versioning tracks changes to feature computation logic. Shared feature repository enables reuse across teams and models.
- 6Data drift monitoring: PSI (Population Stability Index) measures how much a feature distribution has shifted: PSI = Σ (actual% - expected%) × ln(actual%/expected%). PSI < 0.1 = negligible shift, 0.1-0.25 = moderate, >0.25 = significant. KS test gives p-value for distribution shift. Monitor daily or per prediction batch.
- 7Shadow mode and canary deployments: shadow mode = run new model in parallel with old but serve only old model's predictions. Compare outputs to detect bugs before affecting users. Canary = send 1-5% of traffic to new model, monitor metrics, gradually increase to 100% if healthy. Rollback trigger: latency P99 > SLA, error rate spike, primary KPI degradation.
- 8Retraining triggers: (1) scheduled (daily/weekly), (2) data drift detected (PSI > 0.25), (3) model performance degradation (prediction drift, feedback loop detected), (4) new labels available (active learning). Champion-challenger pattern: new model must beat current production model on held-out data before deployment.
Key Formulas
Algorithm / Process
- 1
Step 1 — Containerize: write Dockerfile (base CUDA image, copy requirements.txt, pip install, copy model code, ENTRYPOINT for serving).
- 2
Step 2 — Track experiments: MLflow.start_run(), log_params(config), log_metrics per epoch, log_artifact(model_path). Use mlflow.register_model() for promotion.
- 3
Step 3 — Build serving endpoint: FastAPI app with /predict endpoint, input validation with Pydantic, model loaded at startup (not per-request), batching queue for GPU efficiency.
- 4
Step 4 — Monitor: PSI on input features daily, prediction distribution histogram, error rate per endpoint, latency P50/P95/P99, feedback loop check (does model output affect next day's input?).
- 5
Step 5 — Automate retraining: Airflow/Prefect DAG: data validation → feature engineering → training → evaluation (gate: must exceed production model) → staging deploy → integration tests → canary → full rollout.
| 1 | import mlflow |
| 2 | import mlflow.sklearn |
| 3 | import numpy as np |
| 4 | from sklearn.ensemble import GradientBoostingClassifier |
| 5 | from sklearn.datasets import make_classification |
| 6 | from sklearn.model_selection import train_test_split |
| 7 | from sklearn.metrics import roc_auc_score |
| 8 | |
| 9 | # ─── Training with MLflow tracking ──────────────────────────────────────────── |
| 10 | X, y = make_classification(n_samples=10_000, n_features=20, random_state=42) |
| 11 | X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2) |
| 12 | |
| 13 | mlflow.set_experiment("fraud-detection-v2") |
| 14 | |
| 15 | # Hyperparameter grid (in practice: use Optuna or W&B sweeps) |
| 16 | configs = [ |
| 17 | {"n_estimators": 100, "max_depth": 3, "learning_rate": 0.1}, |
| 18 | {"n_estimators": 200, "max_depth": 4, "learning_rate": 0.05}, |
| 19 | ] |
| 20 | |
| 21 | best_auc, best_run_id = 0, None |
| 22 | |
| 23 | for cfg in configs: |
| 24 | with mlflow.start_run(): |
| 25 | # Log parameters |
| 26 | mlflow.log_params(cfg) |
| 27 | |
| 28 | # Train |
| 29 | model = GradientBoostingClassifier(**cfg, random_state=42) |
| 30 | model.fit(X_train, y_train) |
| 31 | |
| 32 | # Evaluate + log metrics |
| 33 | val_auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1]) |
| 34 | mlflow.log_metric("val_auc", val_auc) |
| 35 | mlflow.log_metric("n_train", len(X_train)) |
| 36 | |
| 37 | # Log model artifact |
| 38 | mlflow.sklearn.log_model(model, "model", |
| 39 | registered_model_name="fraud-detector") |
| 40 | |
| 41 | print(f"Config: {cfg} | Val AUC: {val_auc:.4f}") |
| 42 | if val_auc > best_auc: |
| 43 | best_auc = val_auc |
| 44 | best_run_id = mlflow.active_run().info.run_id |
| 45 | |
| 46 | print(f"Best run: {best_run_id} with AUC={best_auc:.4f}") |
| 47 | |
| 48 | # ─── FastAPI serving endpoint (save as serve.py) ────────────────────────────── |
| 49 | FASTAPI_CODE = """ |
| 50 | from fastapi import FastAPI |
| 51 | from pydantic import BaseModel |
| 52 | import mlflow.sklearn |
| 53 | import numpy as np |
| 54 | |
| 55 | app = FastAPI() |
| 56 | model = mlflow.sklearn.load_model(f"runs:/{run_id}/model") |
| 57 | |
| 58 | class Features(BaseModel): |
| 59 | features: list[float] # expect 20 features |
| 60 | |
| 61 | @app.post("/predict") |
| 62 | def predict(req: Features): |
| 63 | x = np.array(req.features).reshape(1, -1) |
| 64 | prob = model.predict_proba(x)[0, 1] |
| 65 | return {"fraud_probability": float(prob), "is_fraud": bool(prob > 0.5)} |
| 66 | |
| 67 | @app.get("/health") |
| 68 | def health(): |
| 69 | return {"status": "ok"} |
| 70 | """ |
| 71 | print("\nFastAPI serving code:") |
| 72 | print(FASTAPI_CODE) |
| 73 | |
| 74 | # ─── Drift monitoring ───────────────────────────────────────────────────────── |
| 75 | def compute_psi(reference: np.ndarray, current: np.ndarray, |
| 76 | n_bins: int = 10) -> float: |
| 77 | """Population Stability Index for a single feature.""" |
| 78 | bins = np.percentile(reference, np.linspace(0, 100, n_bins + 1)) |
| 79 | bins[0], bins[-1] = -np.inf, np.inf |
| 80 | ref_counts = np.histogram(reference, bins=bins)[0] + 1e-6 |
| 81 | cur_counts = np.histogram(current, bins=bins)[0] + 1e-6 |
| 82 | ref_pct = ref_counts / ref_counts.sum() |
| 83 | cur_pct = cur_counts / cur_counts.sum() |
| 84 | psi = np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct)) |
| 85 | return float(psi) |
| 86 | |
| 87 | # Simulate drift |
| 88 | ref_feature = np.random.randn(10_000) |
| 89 | no_drift = np.random.randn(1_000) # same distribution |
| 90 | drift = np.random.randn(1_000) * 1.5 + 0.8 # shifted |
| 91 | |
| 92 | print(f"PSI (no drift): {compute_psi(ref_feature, no_drift):.4f} (< 0.1 = stable)") |
| 93 | print(f"PSI (with drift): {compute_psi(ref_feature, drift):.4f} (> 0.25 = retrain)") |
Three components: (1) MLflow experiment tracking — logs parameters, metrics, and the serialized model artifact for every training run, with model registration for production promotion; (2) FastAPI serving endpoint template with Pydantic input validation and model loaded at startup (not per-request); (3) PSI drift monitoring implementation that flags when a feature distribution has shifted enough to trigger retraining.
Worked Interview Problems
3 problemsProblem
Your fraud detection model is retrained weekly. Design the full automated retraining pipeline, including what can go wrong and how you handle it.
Solution
Trigger: Airflow DAG runs every Monday at 2am. Also triggered if PSI > 0.25 on any top-5 feature (detected in daily monitoring job).
Data validation: Great Expectations suite checks: schema (expected columns, dtypes), null rates (<5% per column), class balance (>0.5% fraud rate — if lower, label pipeline may be broken), feature ranges (no impossible values).
Training: run on last 90 days data with walk-forward validation. Log all metrics + parameters to MLflow. Fail the DAG if val AUC drops more than 2% vs previous production model.
Gating: champion-challenger test on a held-out 30-day dataset that neither model was trained on. New model must achieve AUC ≥ (current_model_auc - 0.005) AND no regression on precision@1% FPR.
Staging deployment: deploy to staging environment, run integration tests (API contract, latency P99 < 100ms, memory usage). Shadow mode for 24 hours: compare prediction distributions between old and new model on live traffic.
Production rollout: canary 5% for 24 hours → 25% for 24 hours → 100%. Automated rollback if: latency P99 > 150ms, error rate > 0.1%, fraud catch rate drops > 5% vs baseline.
Answer
Pipeline: trigger → data validation → training → champion-challenger gating → staging + shadow → canary → full rollout. Each stage has automated pass/fail criteria. Rollback is automatic if production metrics degrade.
Common Mistakes
Not pinning exact dependency versions in Dockerfile. 'pip install scikit-learn' installs whatever is current — if the version changes between builds, your model may behave differently. Always use requirements.txt with pinned versions (scikit-learn==1.4.0) and rebuild the image when updating.
Loading the model inside the prediction function instead of at startup. Loading a 500MB model on every request adds 2-10 seconds of latency. Load once at application startup, store as a global or in an app-level context.
Using model performance on training data to gate production deployment. The gate must use a held-out test set that was never used in training or hyperparameter tuning. Otherwise the gate is meaningless.
Not handling model unavailability with a fallback. Production models fail. If your serving endpoint has no fallback (rule-based, popularity-based, or previous version), a model crash takes down the entire product feature. Always define a degraded-mode fallback.
Monitoring only model metrics and ignoring data pipeline health. A broken upstream data source (null features, stale data) causes model degradation that looks like model drift but is actually a data issue. Monitor both input data distributions and model metrics.