Applied Scientist
Product Analytics & Business Metrics
Product analytics is the most common non-modeling topic in Applied Scientist and Data Scientist interviews at Meta, Amazon, Airbnb, and Netflix. Every company expects you to define metrics from scratch, diagnose metric anomalies, run funnel analysis, and understand the difference between activity metrics and health metrics. The core skill is translating ambiguous business questions ('is this feature successful?') into precise, measurable, and actionable metric frameworks — and then knowing how those metrics can be gamed or misinterpreted.
Core Theory
- 1Funnel analysis: A conversion funnel tracks users through an ordered sequence of steps toward a goal. Example (e-commerce): Homepage → Search → Product Page → Add to Cart → Checkout → Purchase. At each step, compute: (1) absolute count of users, (2) step conversion rate = users_at_step / users_at_prior_step, (3) overall conversion rate = users_at_bottom / users_at_top. Drop-off identification: the step with the largest absolute and relative drop-off is the highest-leverage optimization target. Attribution: first-touch attribution credits the first marketing channel, last-touch credits the final touch before conversion, multi-touch (linear, time-decay, data-driven/Shapley) distributes credit across all touchpoints. For product funnels, use user-journey analysis to understand which paths lead to conversion vs abandonment — users rarely follow the designed funnel linearly.
- 2Cohort analysis: Groups users by a shared characteristic at a fixed point in time (most commonly, signup week/month) and tracks their behavior over time. Retention curve: for each cohort, plot the fraction of users who return on Day 1, Day 7, Day 30, Day 90. A healthy product shows retention curves that flatten ('smile' shape) rather than continuously declining. D1/D7/D30 retention benchmarks: D1 retention > 40% is good for consumer apps; D30 retention > 20% is considered strong. Cohort heatmaps: rows are cohorts (signup month), columns are time periods (month 0, 1, 2...), cells are retention rates — patterns visible at a glance (improving retention across newer cohorts = product is improving). Cohort-based LTV: compute revenue per cohort over time to project lifetime value.
- 3Engagement metrics — DAU, WAU, MAU and stickiness: Daily Active Users (DAU) = unique users with at least one qualifying action in a calendar day. Weekly Active Users (WAU) = unique users active in a 7-day window. Monthly Active Users (MAU) = unique users active in a 28 or 30-day window. The definition of 'active' matters enormously — 'opened the app' is a weaker signal than 'completed a meaningful action.' Stickiness ratio = DAU / MAU: measures how often monthly active users return daily. Twitter historically ~25%, Facebook ~65%, WhatsApp ~85%. A rising stickiness ratio indicates deepening engagement; a falling ratio indicates users are becoming less habitual. DAU/WAU ratio measures within-week frequency.
- 4North Star Metric and metric decomposition: The North Star Metric (NSM) is a single metric that best captures the value your product delivers to users and is causally linked to long-term revenue. Examples: Airbnb = Nights Booked, Spotify = Time Spent Listening, Slack = Messages Sent, Facebook = DAU. A good NSM is (1) meaningful to users (not just revenue), (2) measurable and unambiguous, (3) actionable (teams can influence it), (4) predictive of long-term business health. Metric decomposition: break the NSM into components that expose the levers. DAU = New Users + Retained Users + Resurrected Users − Churned Users. Revenue = Users × Conversion Rate × Average Order Value × Purchase Frequency. Breaking down a metric reveals which lever is driving changes and which teams own each lever.
- 5User segmentation — RFM and power users: RFM (Recency, Frequency, Monetary) segments users into groups by: R = days since last purchase, F = number of purchases, M = total spend. Score each dimension 1-5, sum scores to segment users into champions (R=5, F=5, M=5), loyal customers, at-risk customers, etc. Power users are the top decile of users by engagement — they generate disproportionate value (on many platforms, the top 10% of users generate 50%+ of engagement). Tracking power user behavior separately from median user behavior prevents the median from masking polarized outcomes (feature helps power users, hurts casual users).
- 6A/B test metric selection — primary, secondary, guardrail: Primary metric: the one metric the experiment is powered to detect and that determines the ship/no-ship decision. Must be measurable within the experiment duration, sensitive to the treatment, and aligned with business value. Secondary metrics: additional metrics to understand the mechanism of the treatment effect (e.g., primary = revenue, secondary = CTR, add-to-cart rate, session length). Guardrail metrics: metrics that must not degrade — they define the acceptable operating envelope. Examples: latency P99 (must stay ≤ 500ms), complaint rate (must not increase > 2%), crash rate. A treatment that improves the primary metric but violates a guardrail is not shippable. North Star vs proxy metric tension: optimizing a proxy metric can improve it while hurting the NSM (more clicks but less revenue per click = worse for the business).
- 7Defining success metrics from scratch (the PM question): Framework — HEART + GSM: (1) Goal: what is the user trying to accomplish? (2) Signals: user behaviors that indicate success or failure. (3) Metrics: the quantifiable measurement of those signals. HEART categories: Happiness (satisfaction scores, NPS), Engagement (sessions, actions per session, depth), Adoption (new users, feature adoption rate), Retention (D30 retention, annual renewal rate), Task Success (completion rate, time-on-task, error rate). When asked 'how do you measure the success of feature X,' start by clarifying: what user problem does X solve? Who is the target user? What does a successful user interaction look like? Then map to: a primary metric capturing user value, secondary metrics for mechanism, guardrails for health.
- 8Revenue metrics — ARPU, LTV, payback period: ARPU (Average Revenue Per User) = Total Revenue / MAU. Differentiates into ARPPU (per paying user) and ARPDAU (per DAU). LTV (Lifetime Value) = ARPU × Average Customer Lifetime = ARPU / Churn Rate (for constant churn). LTV must exceed CAC (Customer Acquisition Cost) for a sustainable business; LTV/CAC > 3 is healthy. Payback period = CAC / Monthly Gross Margin per customer — the time to recoup acquisition cost. Net Revenue Retention (NRR) = revenue from existing customers in period T+1 / revenue from same customers in period T. NRR > 100% (expansion revenue exceeds churn revenue) is the hallmark of a healthy SaaS business.
- 9Common metric anomaly investigation — the 20% drop: When a key metric drops suddenly, use a structured root cause analysis (RCA) framework. Step 1 — Validate: Is the data pipeline working? Check for logging bugs, schema changes, dashboard miscalculations. Step 2 — Scope: When did the drop start? Was it gradual or sudden (sudden = event-driven: deployment, outage; gradual = organic shift)? Is it global or specific to a segment (platform, geography, user cohort)? Step 3 — Decompose: Break the metric into components. If DAU dropped, decompose into new users, retained users, resurrected users, churned users — which component drove the drop? Step 4 — Correlate: Did any deployment, marketing change, or external event coincide? Check the deployment log. Step 5 — Formulate hypotheses and test: Generate 3-5 hypotheses in decreasing likelihood order and test each against data. Step 6 — Fix and validate: After identifying the cause, implement a fix and validate that the metric recovers.
- 10Seasonality effects and normalization: Raw DAU comparisons are misleading without accounting for seasonality. Examples: DAU always drops on weekends for productivity apps; spikes on holidays for e-commerce; spikes in January (New Year's resolutions) for fitness apps. Methods: Year-over-year (YoY) comparison for seasonality-adjusted growth = (current_week_DAU / same_week_last_year_DAU - 1) × 100%. 7-day rolling average smooths day-of-week effects. Decomposing time series into trend + seasonality + residual (STL decomposition, Facebook Prophet) separates the structural trend from cyclical noise. When reporting a metric, always specify whether it is seasonally adjusted, what the comparison baseline is (vs prior period, prior year), and whether the change is statistically significant.
Key Formulas
| 1 | import numpy as np |
| 2 | import pandas as pd |
| 3 | import matplotlib.pyplot as plt |
| 4 | import matplotlib.ticker as mtick |
| 5 | from datetime import datetime, timedelta |
| 6 | |
| 7 | np.random.seed(42) |
| 8 | |
| 9 | # ─── 1. Simulate user event log ─────────────────────────────────────────────── |
| 10 | n_users = 5_000 |
| 11 | start_date = pd.Timestamp('2023-01-01') |
| 12 | |
| 13 | # Each user has a signup date and daily activity probability |
| 14 | users = pd.DataFrame({ |
| 15 | 'user_id': range(n_users), |
| 16 | 'signup_date': pd.to_datetime([ |
| 17 | start_date + timedelta(days=np.random.randint(0, 180)) |
| 18 | for _ in range(n_users) |
| 19 | ]), |
| 20 | # Retention probability decays over time (realistic churn curve) |
| 21 | 'base_retention': np.random.beta(2, 5, n_users), # 0.1 to 0.5 |
| 22 | }) |
| 23 | |
| 24 | # Generate daily activity events |
| 25 | events = [] |
| 26 | for _, user in users.iterrows(): |
| 27 | retention = user['base_retention'] |
| 28 | for day_offset in range(180): |
| 29 | event_date = user['signup_date'] + timedelta(days=day_offset) |
| 30 | if event_date > pd.Timestamp('2023-06-30'): |
| 31 | break |
| 32 | # Retention decays: P(active on day k) = retention^log(k+1) |
| 33 | k = day_offset + 1 |
| 34 | p_active = retention ** (np.log1p(k) / 4) |
| 35 | if np.random.rand() < p_active: |
| 36 | events.append({ |
| 37 | 'user_id': user['user_id'], |
| 38 | 'event_date': event_date, |
| 39 | 'signup_date': user['signup_date'], |
| 40 | }) |
| 41 | |
| 42 | events_df = pd.DataFrame(events) |
| 43 | events_df['days_since_signup'] = ( |
| 44 | events_df['event_date'] - events_df['signup_date'] |
| 45 | ).dt.days |
| 46 | events_df['signup_month'] = events_df['signup_date'].dt.to_period('M') |
| 47 | events_df['event_month'] = events_df['event_date'].dt.to_period('M') |
| 48 | |
| 49 | print(f"Total events: {len(events_df):,}") |
| 50 | print(f"Unique users active: {events_df['user_id'].nunique():,}") |
| 51 | |
| 52 | # ─── 2. Cohort retention matrix ─────────────────────────────────────────────── |
| 53 | def compute_cohort_retention(events: pd.DataFrame, |
| 54 | periods: list[int]) -> pd.DataFrame: |
| 55 | """ |
| 56 | For each signup cohort (by month), compute retention at each day offset. |
| 57 | Returns a heatmap-ready DataFrame: rows=cohort, cols=day offset. |
| 58 | """ |
| 59 | cohort_sizes = (events |
| 60 | .drop_duplicates('user_id')[['user_id', 'signup_month']] |
| 61 | .groupby('signup_month')['user_id'].count() |
| 62 | .rename('cohort_size')) |
| 63 | |
| 64 | rows = [] |
| 65 | for cohort_month, group in events.groupby('signup_month'): |
| 66 | cohort_users = set( |
| 67 | events[events['signup_month'] == cohort_month]['user_id'].unique() |
| 68 | ) |
| 69 | cohort_size = len(cohort_users) |
| 70 | row = {'cohort': str(cohort_month), 'cohort_size': cohort_size} |
| 71 | for day in periods: |
| 72 | window = (group['days_since_signup'] >= day) & (group['days_since_signup'] < day + 7) |
| 73 | retained = group[window]['user_id'].nunique() |
| 74 | row[f'D{day}'] = retained / cohort_size if cohort_size > 0 else np.nan |
| 75 | rows.append(row) |
| 76 | |
| 77 | return pd.DataFrame(rows).set_index('cohort') |
| 78 | |
| 79 | retention_matrix = compute_cohort_retention( |
| 80 | events_df, periods=[0, 1, 7, 14, 30, 60, 90] |
| 81 | ) |
| 82 | print("\nCohort Retention Matrix:") |
| 83 | retention_cols = [c for c in retention_matrix.columns if c.startswith('D')] |
| 84 | print((retention_matrix[retention_cols] * 100).round(1).to_string()) |
| 85 | |
| 86 | # ─── 3. DAU decomposition (new / retained / resurrected / churned) ──────────── |
| 87 | def compute_dau_components(events: pd.DataFrame, |
| 88 | date_col: str = 'event_date', |
| 89 | user_col: str = 'user_id', |
| 90 | signup_col: str = 'signup_date') -> pd.DataFrame: |
| 91 | """ |
| 92 | For each date, decompose DAU into: |
| 93 | - new users (first day active = signup date) |
| 94 | - retained users (active yesterday and today) |
| 95 | - resurrected users (active today but not yesterday, and not new) |
| 96 | - churned users (active yesterday but not today) — shown as negative |
| 97 | """ |
| 98 | daily_active = (events |
| 99 | .groupby([date_col, user_col])[signup_col] |
| 100 | .first() |
| 101 | .reset_index()) |
| 102 | |
| 103 | dates = sorted(daily_active[date_col].unique()) |
| 104 | results = [] |
| 105 | |
| 106 | for i, date in enumerate(dates[1:], 1): |
| 107 | today_users = set(daily_active[daily_active[date_col] == date][user_col]) |
| 108 | yesterday_users = set(daily_active[daily_active[date_col] == dates[i-1]][user_col]) |
| 109 | |
| 110 | # Users whose signup date equals today |
| 111 | new_users_mask = (daily_active[date_col] == date) & (daily_active[signup_col] == date) |
| 112 | new_users = set(daily_active[new_users_mask][user_col]) |
| 113 | |
| 114 | retained = today_users & yesterday_users - new_users |
| 115 | resurrected = today_users - yesterday_users - new_users |
| 116 | churned = yesterday_users - today_users |
| 117 | |
| 118 | results.append({ |
| 119 | 'date': date, |
| 120 | 'dau': len(today_users), |
| 121 | 'new': len(new_users), |
| 122 | 'retained': len(retained), |
| 123 | 'resurrected': len(resurrected), |
| 124 | 'churned': -len(churned), # negative for visualization |
| 125 | }) |
| 126 | |
| 127 | return pd.DataFrame(results) |
| 128 | |
| 129 | dau_components = compute_dau_components(events_df) |
| 130 | weekly = dau_components.set_index('date').resample('W').mean().round(0) |
| 131 | print("\nWeekly average DAU components (sample):") |
| 132 | print(weekly.head(8).to_string()) |
| 133 | |
| 134 | # ─── 4. Funnel analysis ─────────────────────────────────────────────────────── |
| 135 | # Simulate e-commerce funnel for one week |
| 136 | np.random.seed(99) |
| 137 | n_sessions = 50_000 |
| 138 | funnel_stages = { |
| 139 | 'homepage': 50_000, |
| 140 | 'search': int(50_000 * 0.60), # 60% proceed to search |
| 141 | 'product_page': int(50_000 * 0.35), # 35% overall reach product |
| 142 | 'add_to_cart': int(50_000 * 0.12), # 12% |
| 143 | 'checkout': int(50_000 * 0.08), # 8% |
| 144 | 'purchase': int(50_000 * 0.04), # 4% |
| 145 | } |
| 146 | |
| 147 | funnel_df = pd.DataFrame([ |
| 148 | {'stage': stage, 'users': users} |
| 149 | for stage, users in funnel_stages.items() |
| 150 | ]) |
| 151 | funnel_df['step_cvr'] = funnel_df['users'] / funnel_df['users'].shift(1) |
| 152 | funnel_df['overall_cvr'] = funnel_df['users'] / funnel_df['users'].iloc[0] |
| 153 | funnel_df['drop_off'] = funnel_df['users'].shift(1) - funnel_df['users'] |
| 154 | |
| 155 | print("\nFunnel Analysis:") |
| 156 | print(funnel_df.to_string(index=False)) |
| 157 | |
| 158 | # Identify highest-leverage optimization point |
| 159 | max_abs_drop_idx = funnel_df['drop_off'].idxmax() |
| 160 | print(f"\nHighest absolute drop-off: {funnel_df.loc[max_abs_drop_idx, 'stage']}") |
| 161 | print(f" Users lost: {funnel_df.loc[max_abs_drop_idx, 'drop_off']:,.0f}") |
| 162 | |
| 163 | # ─── 5. Seasonality decomposition using rolling averages ────────────────────── |
| 164 | dau_ts = (events_df |
| 165 | .groupby('event_date')['user_id'] |
| 166 | .nunique() |
| 167 | .reset_index(name='dau') |
| 168 | .set_index('event_date') |
| 169 | .sort_index()) |
| 170 | |
| 171 | dau_ts['dau_7d_ma'] = dau_ts['dau'].rolling(7).mean() |
| 172 | dau_ts['dau_28d_ma'] = dau_ts['dau'].rolling(28).mean() |
| 173 | dau_ts['yoy_pct'] = dau_ts['dau'].pct_change(28) * 100 # 4-week change as proxy |
| 174 | |
| 175 | print("\nDAU time series (last 7 days of simulation):") |
| 176 | print(dau_ts.tail(7).round(1).to_string()) |
| 177 | |
| 178 | # ─── 6. Root cause analysis helper: metric decomposition ────────────────────── |
| 179 | def dau_rca(before_df: pd.DataFrame, after_df: pd.DataFrame) -> dict: |
| 180 | """ |
| 181 | Compare DAU components before and after an event to diagnose drops. |
| 182 | Input: DataFrames with columns [new, retained, resurrected, churned]. |
| 183 | """ |
| 184 | before = before_df[['new','retained','resurrected','churned']].mean() |
| 185 | after = after_df[['new','retained','resurrected','churned']].mean() |
| 186 | delta = after - before |
| 187 | pct_change = (delta / before.abs()) * 100 |
| 188 | |
| 189 | result = pd.DataFrame({'before': before, 'after': after, |
| 190 | 'delta': delta, 'pct_change': pct_change}) |
| 191 | # Identify the primary driver |
| 192 | primary_driver = delta.abs().idxmax() |
| 193 | return {'breakdown': result, 'primary_driver': primary_driver, |
| 194 | 'dau_change': after.sum() - before.sum()} |
| 195 | |
| 196 | # Simulate a "before" vs "after" scenario (pretend mid-point is event) |
| 197 | mid = len(dau_components) // 2 |
| 198 | rca_result = dau_rca(dau_components.iloc[:mid], dau_components.iloc[mid:]) |
| 199 | print(f"\nRCA: Primary driver of DAU change = '{rca_result['primary_driver']}'") |
| 200 | print(f"Net DAU change: {rca_result['dau_change']:+.1f} users/day") |
Six building blocks for product analytics interviews: (1) cohort retention matrix showing D0/D1/D7/D30/D60/D90 retention — the canonical way to measure product stickiness; (2) DAU decomposition into new/retained/resurrected/churned — essential for diagnosing DAU drops; (3) funnel analysis with step conversion rates, overall conversion rates, and drop-off identification; (4) seasonality smoothing with 7-day and 28-day rolling averages; (5) RCA helper that identifies the primary driver of a DAU change. In interviews, sketching this decomposition on a whiteboard immediately demonstrates structured thinking.
Worked Interview Problems
3 problemsProblem
You are a Data Scientist at Meta. Your dashboard shows DAU dropped 15% on Monday compared to the same Monday last week. The PM is asking you to explain what happened. Walk through your investigation.
Solution
Step 1 — Validate the data: Before drawing any conclusions, verify the signal is real. Check: (a) is the logging pipeline healthy? Are there any data ingestion delays or missing partitions? (b) Is the dashboard calculation correct? Double-check the SQL query, especially the date filter. (c) Spot-check a secondary data source (e.g., server request logs vs client-side event logs). 30% of 'metric drops' are data pipeline issues, not real product changes.
Step 2 — Characterize the drop — when and where: (a) When did the drop start? Plot hourly DAU — did it start at midnight (suggesting a cron job or deployment) or gradually throughout the day? (b) Is it global or segmented? Break down by platform (iOS, Android, web), geography (US, EMEA, APAC), user cohort (new vs existing users). If it's iOS-only, it's likely an app release issue. If US-only, it could be a US-specific outage or marketing change.
Step 3 — Decompose DAU: Apply the decomposition: DAU = New + Retained + Resurrected - Churned. Did new user acquisition drop (marketing budget cut? acquisition channel outage)? Did retention drop (product bug degrading experience)? Did churn spike? This identifies which team owns the issue.
Step 4 — Check the deployment log: Did any code changes ship in the 48 hours before the drop? Any infrastructure changes (CDN, database migration)? Any marketing campaigns that ended? Any competitor product launch or news event?
Step 5 — Formulate hypotheses in descending probability: (1) Data pipeline issue — most common cause. (2) iOS app update shipped with a bug reducing session starts. (3) Push notification system had an outage — users not reminded to open the app. (4) Major competitor launched a new feature (external). (5) Real organic decline in engagement.
Step 6 — Execute and validate: Test each hypothesis with targeted queries. For each hypothesis, identify what the data pattern would look like if true, and check against actuals. Once root cause is confirmed, estimate the magnitude of impact and timeline for resolution.
Answer
Structure matters more than the answer. The interviewer wants to see: (1) validate data before panicking, (2) characterize (when, who, where), (3) decompose into components, (4) check deployment log / external events, (5) form ranked hypotheses, (6) test each. A common red flag: jumping to a hypothesis without first validating the data pipeline.
Common Mistakes
Reporting averages without segmentation. Average session length, average booking value, and average CTR are all potentially misleading due to Simpson's Paradox and composition effects. Always segment by platform, user type (new vs returning), and geography before drawing conclusions.
Conflating correlation with causation in product metrics. If users who use Feature X have 40% higher retention, this does not mean Feature X causes retention — heavy users naturally engage with more features AND have higher retention. Always ask: is there a self-selection effect? Run an A/B test before attributing causation to a feature adoption correlation.
Not accounting for seasonality in period-over-period comparisons. A 'DAU dropped 10% week-over-week' may be completely normal if the prior week was a holiday week. Always compare to the same period last year, or use a seasonally adjusted metric, before flagging a metric drop as anomalous.
Optimizing proxy metrics that diverge from the North Star. Teams that optimize CTR can increase clickbait that reduces satisfaction and long-term retention. Teams that optimize session length can create addictive features that hurt user well-being. Always check: does improving the proxy metric actually improve the North Star (and can you show this empirically)?
Ignoring the denominator when computing rates. 'Conversions increased 20%' is meaningless without knowing if sessions also increased 20% (flat conversion rate) or stayed flat (genuine conversion rate improvement). Always be explicit about whether you're tracking rates (X / N) or absolutes (X), and what changed in the denominator.