Applied Scientist
RecSys, Ranking & Ads ML
Recommendation systems, search ranking, and ads ML are the business-critical applications at most large tech companies. Applied Scientists in these areas build models that directly drive revenue. You must understand the full pipeline โ from candidate retrieval to ranking to evaluation โ and know the tradeoffs at each stage.
Core Theory
- 1Collaborative filtering: predict a user's preference for an item based on the preferences of similar users (user-based CF) or similar items (item-based CF). Matrix factorization: decompose the user-item interaction matrix R (n_users ร n_items) into U (n_users ร k) and V (n_items ร k) such that R โ UV^T. Each row of U is a user embedding, each row of V is an item embedding. Trained by minimizing squared error on observed ratings + regularization.
- 2Implicit feedback: most real-world data is implicit (clicks, views, purchases) not explicit (star ratings). Problem: we don't know if a user didn't click because they disliked the item or just didn't see it. ALS (Alternating Least Squares) handles this with a confidence weighting: clicks get weight c_ui = 1 + ฮฑ ร r_ui where r_ui is the count of interactions. BPR (Bayesian Personalized Ranking) learns to rank clicked items above unclicked items using pairwise loss.
- 3Two-tower (dual encoder) architecture: user tower (processes user features, history) + item tower (processes item features). Both produce d-dimensional embeddings. Score: dot product or cosine similarity. Trained with softmax over sampled negatives or in-batch negatives. Retrieval: offline compute all item embeddings, build ANN index (FAISS/ScaNN). Online: compute user embedding, query ANN index for top-k items. This is used at YouTube, Spotify, Pinterest, Twitter.
- 4Learning to Rank (LTR): rank candidates given a query/user context. Three paradigms: (1) Pointwise โ score each item independently, optimize MSE or cross-entropy (logistic regression). (2) Pairwise โ for each pair of items, predict which should rank higher. RankNet: p(A > B) = sigmoid(f(A) - f(B)), minimize cross-entropy. LambdaMART: gradient boosting with lambdas derived from NDCG. (3) Listwise โ optimize ranking metrics directly. LambdaRank/SoftMax approaches.
- 5NDCG (Normalized Discounted Cumulative Gain): DCG@k = ฮฃแตข (2^relแตข - 1) / logโ(i+1). IDCG = DCG of ideal ranking. NDCG = DCG/IDCG โ [0,1]. Positions near the top are weighted more (logarithmic discount). Used for evaluating rankings where relevance has degrees (0/1/2/3 relevance labels). MRR (Mean Reciprocal Rank) = 1/rank_of_first_relevant_item, used for navigational search.
- 6Multi-task learning in ranking: a single query can have multiple relevant objectives (maximize CTR AND maximize completion rate AND minimize skip rate). Train one model with multiple heads (CTR head, engagement head, satisfaction head). Loss = weighted sum of individual losses. Challenge: gradient conflicts โ some tasks pull embeddings in different directions. Solutions: Mixture of Experts (MMoE), PCGrad (project conflicting gradients), uncertainty weighting.
- 7Exploration vs exploitation (Explore-exploit): always recommending high-confidence items creates a feedback loop (popular items become more popular). Solutions: (1) epsilon-greedy โ ฮต% random items. (2) Thompson sampling โ sample from posterior distribution of click probability per item, recommend highest sample. (3) Upper Confidence Bound (UCB) โ recommend item with highest (expected reward + exploration bonus). (4) Contextual bandits โ MAB with context features. Lin-UCB, NeuralUCB.
- 8Cold start: new users and new items have no interaction history. User cold start: use demographic features, device, location, and onboarding survey. Item cold start: use item metadata (title, description, category), content-based features, attribute-based collaborative filtering. Fallback: global popularity as a baseline, then transition to personalized as data accumulates. Progressive personalization: linear interpolation between global popularity and personalized model.
Key Formulas
| 1 | import numpy as np |
| 2 | from scipy.sparse import csr_matrix |
| 3 | |
| 4 | np.random.seed(42) |
| 5 | |
| 6 | # โโโ Simulate user-item interaction matrix (implicit feedback) โโโโโโโโโโโโโโโโโ |
| 7 | N_USERS, N_ITEMS = 1000, 5000 |
| 8 | K = 32 # embedding dimension |
| 9 | |
| 10 | # Sparse interactions: each user has interacted with ~20 items |
| 11 | interactions = {} |
| 12 | for u in range(N_USERS): |
| 13 | n_interactions = np.random.randint(5, 40) |
| 14 | items = np.random.choice(N_ITEMS, n_interactions, replace=False) |
| 15 | interactions[u] = set(items) |
| 16 | |
| 17 | # โโโ Matrix Factorization with ALS (Alternating Least Squares) intuition โโโโโโ |
| 18 | class MatrixFactorization: |
| 19 | def __init__(self, n_users, n_items, k=32, lr=0.01, reg=0.01): |
| 20 | scale = 0.01 |
| 21 | self.U = np.random.randn(n_users, k) * scale # user embeddings |
| 22 | self.V = np.random.randn(n_items, k) * scale # item embeddings |
| 23 | self.lr = lr |
| 24 | self.reg = reg |
| 25 | |
| 26 | def predict(self, user_id: int, item_ids: np.ndarray) -> np.ndarray: |
| 27 | return self.U[user_id] @ self.V[item_ids].T |
| 28 | |
| 29 | def train_bpr_step(self, user_id: int, pos_item: int, neg_item: int): |
| 30 | """ |
| 31 | BPR (Bayesian Personalized Ranking) pairwise update. |
| 32 | Learn to rank positive items above negative items. |
| 33 | """ |
| 34 | u = self.U[user_id] |
| 35 | v_pos = self.V[pos_item] |
| 36 | v_neg = self.V[neg_item] |
| 37 | |
| 38 | # Predicted score difference |
| 39 | x_uij = u @ v_pos - u @ v_neg |
| 40 | |
| 41 | # Sigmoid gradient (BPR loss derivative) |
| 42 | sigmoid_grad = -1 / (1 + np.exp(x_uij)) # d/dx [-log sigma(x)] |
| 43 | |
| 44 | # Gradient updates |
| 45 | self.U[user_id] -= self.lr * (sigmoid_grad * (v_pos - v_neg) + self.reg * u) |
| 46 | self.V[pos_item] -= self.lr * (sigmoid_grad * u + self.reg * v_pos) |
| 47 | self.V[neg_item] -= self.lr * (-sigmoid_grad * u + self.reg * v_neg) |
| 48 | |
| 49 | return float(-np.log(1 / (1 + np.exp(-x_uij)))) # BPR loss |
| 50 | |
| 51 | def recommend(self, user_id: int, k: int = 10, |
| 52 | exclude_seen: bool = True) -> np.ndarray: |
| 53 | """Get top-k item recommendations for a user.""" |
| 54 | all_scores = self.U[user_id] @ self.V.T |
| 55 | if exclude_seen: |
| 56 | seen = list(interactions.get(user_id, set())) |
| 57 | all_scores[seen] = -np.inf |
| 58 | return np.argpartition(all_scores, -k)[-k:] |
| 59 | |
| 60 | # โโโ Training loop โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 61 | model = MatrixFactorization(N_USERS, N_ITEMS, k=K) |
| 62 | all_items = set(range(N_ITEMS)) |
| 63 | total_loss = 0.0 |
| 64 | |
| 65 | for step in range(10_000): |
| 66 | u = np.random.randint(N_USERS) |
| 67 | if not interactions[u]: |
| 68 | continue |
| 69 | pos = np.random.choice(list(interactions[u])) |
| 70 | # Sample negative item (not seen by user) |
| 71 | neg = np.random.randint(N_ITEMS) |
| 72 | while neg in interactions[u]: |
| 73 | neg = np.random.randint(N_ITEMS) |
| 74 | total_loss += model.train_bpr_step(u, pos, neg) |
| 75 | |
| 76 | if (step + 1) % 2000 == 0: |
| 77 | print(f"Step {step+1}: avg BPR loss = {total_loss/(step+1):.4f}") |
| 78 | |
| 79 | # โโโ Evaluation: Recall@10 โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 80 | def recall_at_k(model, users_to_eval: int = 100, k: int = 10) -> float: |
| 81 | hits = 0 |
| 82 | for u in range(users_to_eval): |
| 83 | if len(interactions[u]) < 2: |
| 84 | continue |
| 85 | # Hold out last item as test |
| 86 | test_item = list(interactions[u])[-1] |
| 87 | recs = model.recommend(u, k=k) |
| 88 | if test_item in recs: |
| 89 | hits += 1 |
| 90 | return hits / users_to_eval |
| 91 | |
| 92 | recall = recall_at_k(model, users_to_eval=200, k=10) |
| 93 | print(f"\nRecall@10: {recall:.4f} (random baseline: {10/N_ITEMS:.4f})") |
| 94 | |
| 95 | # โโโ NDCG calculation โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 96 | def ndcg_at_k(relevances: list, k: int) -> float: |
| 97 | """Compute NDCG@k given a list of relevance scores for ranked items.""" |
| 98 | dcg = sum((2**rel - 1) / np.log2(i + 2) for i, rel in enumerate(relevances[:k])) |
| 99 | ideal = sorted(relevances, reverse=True) |
| 100 | idcg = sum((2**rel - 1) / np.log2(i + 2) for i, rel in enumerate(ideal[:k])) |
| 101 | return dcg / idcg if idcg > 0 else 0.0 |
| 102 | |
| 103 | # Example: model ranked [relevant, irrelevant, relevant, irrelevant, relevant] |
| 104 | relevances = [1, 0, 1, 0, 1, 0, 0, 1, 0, 0] |
| 105 | print(f"NDCG@5: {ndcg_at_k(relevances, 5):.4f}") |
| 106 | print(f"NDCG@10: {ndcg_at_k(relevances, 10):.4f}") |
Implements matrix factorization with BPR pairwise loss โ the standard for implicit feedback recommendation. BPR update: for each (user, pos_item, neg_item) triple, increase the predicted score of pos_item above neg_item using sigmoid gradient. Includes Recall@10 evaluation (did the model include the held-out item in top-10?) and NDCG@k computation from relevance score lists.
Worked Interview Problems
3 problemsProblem
Design a recommendation system to generate a 30-song weekly playlist personalized for each of Spotify's 600M users. Walk through your complete architecture.
Solution
Two-stage pipeline: (1) Candidate retrieval โ narrow 100M+ tracks to ~500 candidates per user. (2) Ranking โ score 500 candidates with a richer model to select the final 30.
Retrieval: train a two-tower model. User tower: embeds listening history (sequence of tracks โ attention aggregation), recent moods (skips/repeats ratio), demographic context. Track tower: embeds audio features (tempo, energy, acousticness from Spotify Audio Features API), genre, era, artist graph embedding. Trained with in-batch softmax negatives. Offline: compute all 100M track embeddings, build FAISS index (updated weekly). Online: compute user embedding, top-500 via ANN.
Ranking: features โ user embedding ร track embedding (dot product and element-wise product), user's historical preference for track's attributes (tempo, energy), time-since-last-heard (avoid recent repeats), collaborative signal (users similar to you liked this track this week). Model: LightGBM or a 3-layer MLP. Trained on implicit labels: streams > 30 seconds = positive, skip < 10 seconds = negative.
Diversity: a pure relevance-maximizing ranker picks very similar tracks. Add a diversity re-ranking step: greedy algorithm that maximizes relevance while ensuring tracks span multiple genres, tempos, and eras (Maximal Marginal Relevance). Constraint: no more than 3 tracks per artist.
Evaluation: offline โ AUC-PR, NDCG@30, diversity score (entropy of genre distribution). Online A/B test โ stream rate, skip rate, session completion rate, 30-day return rate (does the user listen again next week?). Monitor: new track exposure rate (did we recommend any new artists?).
Answer
Two-tower retrieval (500 candidates) โ LightGBM ranker โ diversity re-ranking (MMR) โ final 30. Evaluation: NDCG@30 offline, skip rate + 30-day return rate online. Weekly batch update cycle with daily user embedding refresh for recent listening signal.
Common Mistakes
Optimizing CTR as the sole ranking objective. CTR is correlated with satisfaction but not identical. Clickbait items increase CTR while reducing long-term engagement and user trust. Always use CTR as one signal among several: completion rate, share rate, negative feedback rate.
Ignoring the explore-exploit tradeoff. A purely exploitation model converges to recommending the same popular items to everyone, starving new items of exposure and reducing catalog diversity. This creates a harmful feedback loop. Explicitly design for exploration.
Not handling the position bias in training data. Items shown in position 1 have higher CTR than position 10 even if position-10 items are equally relevant. Training without correcting for position bias teaches the model 'items shown high = better' rather than 'items are inherently better.' Use inverse propensity scoring or position-aware modeling.
Evaluating recommendation systems only offline. Offline metrics (NDCG, AUC-PR) are necessary but not sufficient โ the test distribution is biased toward items already recommended by the previous system. Online A/B testing is required to measure true improvement.
Not considering latency when designing the ranking model. A ranking model with 500ms inference time will fail the <100ms SLA for most recommendation systems. Always design with the latency budget in mind: retrieval + ranking + post-processing must fit within the total budget.