Applied Scientist
Research Communication & Problem Framing
Applied Scientists must translate complex analyses into decisions that PMs, engineers, and executives can act on. This requires structured thinking: defining the right question before analyzing data, choosing metrics that measure what matters, communicating uncertainty clearly, and writing memos that drive alignment. This skill is often the difference between a strong hire and a no-hire at senior levels.
Core Theory
- 1Problem framing before analysis: the most common Applied Scientist mistake is jumping to analysis before defining the right question. Framework: (1) What is the business question? (2) What is the ML objective that maps to it? (3) What data would answer this question? (4) What are the success criteria? (5) What are the failure modes? Spending 20% of your time on framing saves 80% of wasted analysis.
- 2Metric hierarchy: north-star metric (long-term value, e.g., 12-month LTV), driver metrics (predictive of north-star, e.g., 7-day retention), proxy metrics (measurable in experiments, e.g., CTR, completion rate), guardrail metrics (must not degrade, e.g., latency, diversity, safety). Experiments optimize proxy metrics; decisions should be evaluated against north-star metrics. Goodhart's Law: 'When a measure becomes a target, it ceases to be a good measure.'
- 3Audience-appropriate communication: PM needs to know โ did the feature work, should we ship it? Answer: 'Yes, CTR improved 8% (statistically significant), retention improved 3% (directionally positive), no guardrail regressions. Recommend shipping.' Engineer needs to know โ how does the system work and what's the latency? Exec needs to know โ business impact and next steps: '$2.4M annual revenue increase, team needs 3 months to implement the next iteration.'
- 4Uncertainty communication: always state confidence intervals, not just point estimates. Never say 'CTR will increase 8%' โ say 'CTR improved 8% ยฑ 2% (95% CI) in our test of 500k users over 14 days.' PMs and executives must understand uncertainty to make good decisions. Communicating overconfidence and then being wrong destroys trust.
- 5Navigating ambiguity: when asked an open-ended question, resist the urge to start analyzing immediately. Ask: (1) What decision are we trying to make? (2) What would change our conclusion? (3) What is the minimum analysis needed? (4) What timeframe do we have? Answering the right question roughly is better than answering the wrong question precisely.
- 6Root cause vs. correlation: 'Feature X correlates with higher retention' is a descriptive finding. 'Users who use Feature X have higher retention because X helps them complete their core job-to-be-done' is a causal claim that requires stronger evidence. Always be explicit about whether you're reporting correlation or causation, and what evidence would be needed to establish causality.
- 7Research memo structure: (1) Summary (2-3 sentences โ answer first, then evidence). (2) Business context (why does this question matter?). (3) Methods (what analysis, what data, what time period, what caveats). (4) Results (key finding with quantification and uncertainty). (5) Interpretation (what does this mean for the product?). (6) Recommendations (specific, actionable next steps with owners and timelines). (7) Limitations and next steps. Writing well is tested at Amazon (PR/FAQ), Google (design docs), and Meta (notes culture).
- 8Avoiding analysis errors: (1) Survivorship bias โ only analyzing users who stayed. (2) Simpson's paradox โ aggregate trend opposite to every subgroup trend (usually due to confounding). (3) Cherry-picking metrics โ reporting the one metric that improved while ignoring regressions. (4) P-hacking โ running many analyses and reporting only the significant one. (5) Neglecting base rates โ a 10% lift on a metric that starts at 0.5% is still only 0.55%.
Key Formulas
| 1 | import numpy as np |
| 2 | import pandas as pd |
| 3 | from scipy import stats |
| 4 | from typing import Dict, Tuple |
| 5 | |
| 6 | np.random.seed(42) |
| 7 | |
| 8 | # โโโ Simulate: retention dropped 8% last week โ investigate โโโโโโโโโโโโโโโโโโโ |
| 9 | # Simulate user data with multiple dimensions |
| 10 | |
| 11 | N = 50_000 |
| 12 | platforms = np.random.choice(['iOS', 'Android', 'Web'], N, p=[0.4, 0.35, 0.25]) |
| 13 | regions = np.random.choice(['US', 'EU', 'APAC'], N, p=[0.5, 0.3, 0.2]) |
| 14 | user_ages = np.random.choice(['new', 'returning', 'veteran'], N, p=[0.2, 0.5, 0.3]) |
| 15 | |
| 16 | # Simulate: Android in EU had a bug last week โ low retention for this segment |
| 17 | def retention_prob(platform, region, user_age, is_last_week): |
| 18 | base = 0.70 |
| 19 | if user_age == 'new': base -= 0.15 |
| 20 | if user_age == 'veteran': base += 0.10 |
| 21 | if is_last_week and platform == 'Android' and region == 'EU': |
| 22 | base -= 0.30 # bug introduced this week |
| 23 | return np.clip(base + np.random.randn() * 0.05, 0, 1) |
| 24 | |
| 25 | # This week and last week data |
| 26 | retained_this = np.array([retention_prob(p, r, a, True) > 0.5 |
| 27 | for p, r, a in zip(platforms, regions, user_ages)]) |
| 28 | retained_prior = np.array([retention_prob(p, r, a, False) > 0.5 |
| 29 | for p, r, a in zip(platforms, regions, user_ages)]) |
| 30 | |
| 31 | df = pd.DataFrame({ |
| 32 | 'platform': platforms, 'region': regions, 'user_age': user_ages, |
| 33 | 'retained_this': retained_this, 'retained_prior': retained_prior |
| 34 | }) |
| 35 | |
| 36 | # โโโ Step 1: Headline metric โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 37 | overall_delta = df.retained_this.mean() - df.retained_prior.mean() |
| 38 | print(f"Overall retention delta: {overall_delta*100:.2f}pp") |
| 39 | |
| 40 | # โโโ Step 2: Segment drill-down โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 41 | print(" |
| 42 | Retention delta by platform ร region:") |
| 43 | print(f"{'Segment':<20} | {'This week':>10} | {'Prior week':>10} | {'Delta':>8} | {'N':>7}") |
| 44 | print("-" * 65) |
| 45 | |
| 46 | for platform in ['iOS', 'Android', 'Web']: |
| 47 | for region in ['US', 'EU', 'APAC']: |
| 48 | mask = (df.platform == platform) & (df.region == region) |
| 49 | if mask.sum() < 100: |
| 50 | continue |
| 51 | sub = df[mask] |
| 52 | delta = sub.retained_this.mean() - sub.retained_prior.mean() |
| 53 | # Highlight significant drops |
| 54 | flag = " โ INVESTIGATE" if delta < -0.10 else "" |
| 55 | print(f"{platform+'/'+region:<20} | {sub.retained_this.mean():>10.3f} | " |
| 56 | f"{sub.retained_prior.mean():>10.3f} | {delta:>+8.3f}{flag}") |
| 57 | |
| 58 | # โโโ Step 3: Statistical significance of the Android/EU drop โโโโโโโโโโโโโโโโโ |
| 59 | mask_bug = (df.platform == 'Android') & (df.region == 'EU') |
| 60 | sub_bug = df[mask_bug] |
| 61 | t_stat, p_val = stats.ttest_ind(sub_bug.retained_this.astype(float), |
| 62 | sub_bug.retained_prior.astype(float)) |
| 63 | print(f" |
| 64 | Android/EU retention drop: t={t_stat:.2f}, p={p_val:.4f}") |
| 65 | print(f"Effect size: {(sub_bug.retained_this.mean()-sub_bug.retained_prior.mean())*100:.1f}pp") |
| 66 | print(f"Affected users: {mask_bug.sum():,}") |
| 67 | |
| 68 | # โโโ Step 4: Business impact quantification โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 69 | dau = 5_000_000 |
| 70 | eu_android_fraction = 0.35 * 0.10 # 35% EU ร 10% Android of total users (estimate) |
| 71 | affected_users = int(dau * eu_android_fraction) |
| 72 | retention_drop = 0.30 |
| 73 | lost_daily_actives = affected_users * retention_drop |
| 74 | print(f" |
| 75 | Business impact estimate:") |
| 76 | print(f" Affected users: ~{affected_users:,}") |
| 77 | print(f" Lost daily actives: ~{int(lost_daily_actives):,}") |
| 78 | monthly_at_risk = lost_daily_actives * 50 / 365 * 30 |
| 79 | print(f" Monthly revenue at risk: ~USD {monthly_at_risk:.0f}") |
Structured investigation of a metric drop: (1) headline metric โ overall delta, (2) segment drill-down by platform ร region to isolate root cause, (3) statistical significance test on the affected segment, (4) business impact quantification. This is the exact analysis structure an Applied Scientist is expected to produce when a PM asks 'retention dropped 8% last week.'
Worked Interview Problems
3 problemsProblem
At 9am Monday, a PM messages you: 'Our 7-day retention dropped 8% last week compared to the prior week. What happened?' Walk through your investigation.
Solution
Before touching data, clarify the question: 'When you say 7-day retention, do you mean users who return within 7 days of their first action, or users who are active at day 7? And is this 8% relative (e.g., 40% โ 36.8%) or 8 percentage points (40% โ 32%)? And compared to which prior week โ same week last year or the immediately preceding week?'
Check for external events: was there a product deployment last week? A marketing campaign? A holiday or seasonal event? Check the incident log and engineering deploys. A deployment on Wednesday that coincides with the drop onset is your first lead.
Segment the metric: break down by platform (iOS/Android/Web), geography, user cohort (new vs. returning), acquisition channel. A drop that affects only Android users points to an Android-specific bug. A drop only in new users points to onboarding.
Check data pipeline: is the measurement itself broken? Is the event logging working? A 'drop' caused by events being dropped from the logging pipeline is a false alarm. Check event counts and coverage rates.
Form hypotheses and test: 'Hypothesis 1: Android v4.2 release had a crash affecting the main retention action. Evidence: crash rate in Android went from 0.1% to 2.3% on Tuesday.' Test each hypothesis with data.
Communicate findings: 'We identified a crash bug in Android v4.2 affecting 350,000 users in the EU. Estimated 150,000 users failed to complete their retention action due to the crash. This accounts for 6.2pp of the 8pp drop. The remaining 1.8pp is likely seasonal (same week last year also showed slight decline). Recommending: hot fix deployment for Android, targeted re-engagement push notification to affected users.'
Answer
Investigation framework: (1) clarify the metric definition, (2) check for external events/deployments, (3) segment the metric by platform, geo, cohort, (4) verify data pipeline integrity, (5) form and test specific hypotheses, (6) communicate: root cause + affected users + business impact + specific action items.
Common Mistakes
Conflating 'not statistically significant' with 'no effect.' An underpowered experiment with p=0.12 and 95% CI of [-1%, 5%] is consistent with a real 3% effect. Saying 'the experiment showed no effect' is incorrect and misleads decision-makers.
Reporting results without communicating uncertainty. 'CTR increased 8%' without 'ยฑ 2% (95% CI) with N=500k, p=0.001, 14-day duration' leaves decision-makers without the information they need to assess confidence.
Jumping to analysis without clarifying the business question. You can build a beautiful model that answers the wrong question. Spend 20 minutes defining the right question before writing any code.
Cherry-picking the metric that moved in the right direction. If you test 10 metrics and one shows significance, that one significant result has a 5% chance of being a false positive if ฮฑ=0.05. Report all pre-specified metrics, not just the positive ones.
Using technical jargon with non-technical stakeholders without translation. Saying 'the model has AUC=0.91' to a PM is meaningless. Say: 'The model correctly identifies high-risk customers 91% of the time it ranks them above low-risk customers โ much better than chance (50%) and better than our current rule-based system (78%).'