ML Engineer
ML System Design
ML system design is the most differentiating interview round for senior MLE roles. You are given an open-ended problem ('design a recommendation system for Netflix') and must architect the full ML pipeline โ from data collection to serving to monitoring โ while navigating latency, scale, and correctness tradeoffs.
Core Theory
- 1Problem framing: before any architecture, define (1) the ML objective (maximize CTR? engagement? revenue?), (2) the proxy metric (what the model optimizes), (3) the business KPI (what actually matters), and (4) the constraints (latency SLA, throughput, accuracy floor). Misaligning proxy metric and KPI is the #1 production ML failure.
- 2Two-stage recommendation pipeline: (1) Candidate retrieval โ narrow billions of items to thousands using approximate nearest neighbor (ANN) search on user/item embeddings. Fast but approximate. (2) Ranking โ score thousands of candidates with a heavier model (GBDT or NN) to produce a final ranked list. Used at YouTube, Netflix, Spotify, LinkedIn.
- 3Feature stores: offline store (batch-computed features for training, e.g., Hive/BigQuery) + online store (low-latency serving features, e.g., Redis/DynamoDB). The split prevents train-serve skew โ features used at serving time must exactly match features used at training time. Feast and Tecton are the main frameworks.
- 4Model serving architectures: synchronous REST/gRPC (simple, high latency), async batch inference (high throughput, delayed), streaming inference (Kafka โ model โ response). For <50ms SLA: use model optimization (quantization, ONNX) + batching on GPU + warm replicas with Kubernetes.
- 5A/B testing for models: split traffic into control (old model) and treatment (new model) groups. Measure primary metric (CTR, revenue, engagement) and guardrail metrics (latency, diversity, complaints). Need sufficient sample size (power analysis) and hold for novelty effects (at least 1-2 weeks).
- 6Data flywheel: user interactions โ labeled training data โ better model โ better predictions โ more interactions. The most defensible ML systems have strong data flywheels. Design systems to maximize label quality and volume.
- 7Monitoring in production: data drift (input distribution shift โ PSI, KL divergence, KS test), concept drift (relationship between X and y changes), prediction drift (output distribution shift), and feedback loop detection (model affects the very data it trains on). Set up automated alerts on all four.
- 8Cold start problem: new users have no history โ can't personalize. Solutions: (1) content-based filtering using item metadata, (2) demographic priors, (3) exploration policy (epsilon-greedy or Thompson sampling), (4) onboarding survey to collect initial preferences.
Key Formulas
Algorithm / Process
- 1
Step 1 โ Clarify requirements: scale (users, items, QPS), latency SLA, freshness requirements, personalization depth.
- 2
Step 2 โ Define ML objective: binary CTR prediction? Ranking? Session-level optimization? Multi-task (CTR + completion + like)?
- 3
Step 3 โ Data pipeline: what data exists (user history, item metadata, context)? How is it collected, stored, and refreshed?
- 4
Step 4 โ Feature engineering: offline (batch) features via Spark/BigQuery + online (real-time) features from Redis. Define feature store schema.
- 5
Step 5 โ Model architecture: retrieval (two-tower embeddings + ANN) โ ranking (GBDT or DCN for tabular, transformer for sequential). Justify each choice.
- 6
Step 6 โ Training infrastructure: distributed training (DDP), data pipeline (TFRecord/Parquet), experiment tracking (MLflow/W&B), model registry.
- 7
Step 7 โ Serving: latency budget breakdown (retrieval Xms, ranking Yms, post-processing Zms). Caching strategy. Fallback for model unavailability.
- 8
Step 8 โ Evaluation and monitoring: offline metrics (AUC, NDCG), online metrics (CTR, engagement), A/B test design, drift alerts, retraining triggers.
| 1 | import numpy as np |
| 2 | |
| 3 | # โโโ Simplified Two-Tower Architecture โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 4 | # In production: user tower and item tower are neural networks. |
| 5 | # Here we sketch the inference + ANN retrieval logic. |
| 6 | |
| 7 | np.random.seed(42) |
| 8 | D = 64 # embedding dimension |
| 9 | N_USERS = 1000 |
| 10 | N_ITEMS = 100_000 |
| 11 | |
| 12 | # Simulate trained embeddings (in practice: output of user/item tower NNs) |
| 13 | user_embeddings = np.random.randn(N_USERS, D).astype(np.float32) |
| 14 | item_embeddings = np.random.randn(N_ITEMS, D).astype(np.float32) |
| 15 | |
| 16 | # Normalize for cosine similarity (dot product on unit vectors = cosine) |
| 17 | user_embeddings /= np.linalg.norm(user_embeddings, axis=1, keepdims=True) |
| 18 | item_embeddings /= np.linalg.norm(item_embeddings, axis=1, keepdims=True) |
| 19 | |
| 20 | # โโโ Exact retrieval (for small N_ITEMS) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 21 | def exact_retrieval(user_emb: np.ndarray, item_embs: np.ndarray, k: int = 100): |
| 22 | """Retrieve top-k items for a user. O(N_ITEMS * D).""" |
| 23 | scores = item_embs @ user_emb # (N_ITEMS,) dot products |
| 24 | top_k_idx = np.argpartition(scores, -k)[-k:] |
| 25 | top_k_idx = top_k_idx[np.argsort(scores[top_k_idx])[::-1]] |
| 26 | return top_k_idx, scores[top_k_idx] |
| 27 | |
| 28 | user_id = 42 |
| 29 | top_items, top_scores = exact_retrieval(user_embeddings[user_id], item_embeddings) |
| 30 | print(f"Top-5 items for user {user_id}: {top_items[:5]}") |
| 31 | print(f"Scores: {top_scores[:5].round(3)}") |
| 32 | |
| 33 | # โโโ ANN with FAISS (production approach) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 34 | # pip install faiss-cpu |
| 35 | # import faiss |
| 36 | # index = faiss.IndexFlatIP(D) # Inner Product (cosine on normalized vecs) |
| 37 | # index = faiss.IndexIVFFlat(...) # Inverted file index: faster, approximate |
| 38 | # index.add(item_embeddings) |
| 39 | # distances, indices = index.search(user_embeddings[user_id:user_id+1], k=100) |
| 40 | # Retrieval: ~1ms for 1M items vs ~100ms for exact search |
| 41 | |
| 42 | # โโโ Ranking stage (GBDT scoring the top-100 candidates) โโโโโโโโโโโโโโโโโโโโโ |
| 43 | # Features: user features + item features + cross features (user x item interaction) |
| 44 | # Model: LightGBM with features [user_age, item_category, past_ctr, recency, ...] |
| 45 | # Output: P(click) for each candidate โ re-rank by score |
| 46 | |
| 47 | def mock_ranker(candidate_item_ids, user_id, n_features=20): |
| 48 | """Simulate GBDT ranker output.""" |
| 49 | n = len(candidate_item_ids) |
| 50 | features = np.random.randn(n, n_features) # in practice: feature store lookup |
| 51 | scores = 1 / (1 + np.exp(-features.mean(axis=1))) # mock sigmoid output |
| 52 | ranked_idx = np.argsort(scores)[::-1] |
| 53 | return candidate_item_ids[ranked_idx], scores[ranked_idx] |
| 54 | |
| 55 | final_items, final_scores = mock_ranker(top_items, user_id) |
| 56 | print(f"\nFinal ranked top-5: {final_items[:5]}") |
| 57 | print(f"P(click): {final_scores[:5].round(3)}") |
Implements the two-stage retrieval-ranking pipeline used at Netflix, YouTube, and Spotify. Stage 1: exact dot-product retrieval (in production, replaced with FAISS IVF index for ~1ms retrieval from 100M items). Stage 2: a heavier GBDT ranker rescores the top-100 candidates using richer features. The architecture comment shows how FAISS would be used in production.
Worked Interview Problems
3 problemsProblem
Design YouTube's video recommendation system from scratch. Walk through your full architecture.
Solution
Clarify: 2B users, 500 hours of video uploaded/min, 1B hours watched/day. Goal: maximize long-term user satisfaction (not just CTR โ watch time, diversity, no rabbit holes). Latency: <200ms for recommendation page load.
Two-stage pipeline: Candidate Generation (retrieval) โ Ranking. Retrieval narrows 500M videos to 100-200 candidates per user using user history embeddings. Ranking scores the 200 candidates with a deep neural network.
Retrieval: train a two-tower model โ user tower (watch history sequence, demographics, context) + video tower (title, description, category, view count). Train with softmax over all videos (negative sampling). Serve with FAISS ANN index. Rebuild index daily.
Ranking model: DCN (Deep & Cross Network) v2 or a transformer-based model. Features: userรvideo cross features (has user watched this channel? user's average watch time for this category), freshness (prefer newer videos), diversity (penalize same channel in top-10).
Training data: implicit labels (watch time / video length > 0.5 = positive). Multi-task: predict watch%, like, subscribe simultaneously. Use progressive training โ retrain on last 7 days daily.
Serving: retrieval ~10ms (FAISS), ranking ~50ms (GPU batch inference via Triton), post-processing (de-duplication, diversity re-ranking, freshness boost) ~5ms. Total ~65ms well within budget.
Monitoring: data drift on watch-time distribution, CTR drift, diversity KPI (unique channels in top-10), filter bubble detection (user watch history entropy).
Answer
Two-stage: FAISS-based two-tower retrieval โ DCN ranking with multi-task (watch%, like, subscribe) training. Feature store for low-latency features. Daily retraining. Monitor CTR, watch time, diversity, and filter bubble metrics.
Common Mistakes
Jumping to model architecture before defining the ML objective. The most common system design failure is solving the wrong problem โ optimizing CTR when the business cares about retention. Always spend 5 minutes on problem framing.
Designing for perfect latency at the expense of everything else. A system that can't retrieve in <10ms at 100k QPS is useless. Always do a latency budget breakdown: retrieval + ranking + serving overhead must fit the SLA.
Forgetting train-serve skew. If the feature pipeline for training uses different logic than the serving feature store, model performance in production will always be worse than offline. Feature stores exist to solve this.
Not addressing data cold start. 'New users get global popularity' is a valid starting answer, but then explain the evolution: demographic priors โ item-based CF โ gradually incorporating their history.
No monitoring plan. Every system design answer needs: what metrics do you monitor, what triggers a retraining, what triggers a rollback, what happens if the model is unavailable (fallback strategy).