Research Scientist
Research Vision & Novel Ideas
Research Scientist roles require you to generate, evaluate, and pitch original research ideas. Interviewers at OpenAI, Anthropic, DeepMind, and Meta AI ask: 'What would you work on if you joined our team?' Your answer reveals scientific taste, knowledge of the field, and research maturity. Vague answers are a red flag โ specific, falsifiable hypotheses with concrete experiments are what distinguishes top candidates.
Core Theory
- 1Where ideas come from: (1) Gaps in existing work โ after reading a paper, ask 'what question does this leave open?' or 'what assumption does this rely on that might be wrong?' (2) Analogies across fields โ a technique that works in computer vision may work in NLP with modification. (3) Failure modes โ why does method X fail on distribution Y? The failure mode often points directly to an improvement. (4) Practical constraints โ what prevents real-world deployment of this research? Solving the constraint is a paper.
- 2Hypothesis formation: a good research hypothesis is (1) specific and falsifiable โ 'adding auxiliary consistency loss will improve factual accuracy of LLMs by 10-20% on TruthfulQA without degrading generation quality', NOT 'we can improve LLMs.' (2) Grounded in theory โ there should be a principled reason to believe it. (3) Minimal experiment โ what is the smallest experiment that could confirm or refute this? (4) Clear success criterion โ what result would constitute a publishable paper?
- 3Research agenda development: a senior Research Scientist doesn't just execute projects โ they identify a research direction with 3-5 connected papers. Ask: (1) What is the fundamental problem? (2) What is the minimal version that could be demonstrated? (3) What is the full solution? (4) What are the related problems that would benefit from solving this? A good research agenda compounds โ each paper enables the next.
- 4Scientific taste: the ability to distinguish important problems from minor ones. Hallmarks: (1) works on problems that would change the field if solved, not incremental improvements, (2) chooses methods for principled reasons, not because they're trendy, (3) can identify when a 'new' paper is actually a rediscovery of prior work, (4) knows which claims require more evidence, (5) prefers simple explanations (Occam's razor).
- 5What makes a compelling research contribution: (1) solves a problem that was previously unsolved or unknown, (2) provides a new perspective on a well-studied problem, (3) unifies previously disconnected observations, (4) introduces a new benchmark or evaluation that changes how the field measures progress, (5) makes a practical system work that didn't before. 'Beats SOTA by 1%' is the weakest form; 'changes how we think about X' is the strongest.
- 6Risk assessment for research ideas: every idea should be evaluated on (1) impact โ what changes if this works?, (2) probability of success โ is there a plausible mechanism?, (3) cost โ how much time/compute is needed for the minimal experiment?, (4) differentiation โ is this already being worked on by 10 groups? High-impact, plausible, cheap, and differentiating ideas are worth pursuing first.
- 7Knowing what's been tried: before proposing an idea, search arXiv for related work from the last 3 years. Use Semantic Scholar to find papers citing the key work. A common interview mistake: proposing an idea that was published 6 months ago. Say: 'I'm not aware of prior work on this specific combination โ if you are, I'd love to discuss what they found.'
- 8Staying current: follow arXiv.org/cs.LG and cs.CL daily (or use arxiv-sanity.org). Read the top 5-10 papers from NeurIPS/ICLR/ICML proceedings. Follow key researchers on Twitter/X (Andrej Karpathy, Yann LeCun, Ilya Sutskever, Percy Liang, Chelsea Finn). Read team research pages before interviewing โ know the 5 most recent papers from the team you'd join.
Key Formulas
| 1 | # โโโ Research Idea Evaluation Framework โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 2 | |
| 3 | IDEA_EVALUATION_TEMPLATE = """ |
| 4 | RESEARCH IDEA EVALUATION CHECKLIST |
| 5 | |
| 6 | Title: [Concise description of the idea] |
| 7 | |
| 8 | 1. PROBLEM STATEMENT |
| 9 | - What specific problem does this solve? |
| 10 | - Why has it not been solved before? |
| 11 | - What would change in the field if this worked? |
| 12 | |
| 13 | 2. HYPOTHESIS |
| 14 | - Specific claim: [exact testable statement] |
| 15 | - Theoretical grounding: [why should this work?] |
| 16 | - Null hypothesis: [what would we see if it doesn't work?] |
| 17 | |
| 18 | 3. MINIMAL EXPERIMENT |
| 19 | - Dataset/benchmark: [which existing benchmark tests this?] |
| 20 | - Baseline: [what is the current best method we compare to?] |
| 21 | - Method: [what exactly would we implement?] |
| 22 | - Success criterion: [what number/result = publishable?] |
| 23 | - Time estimate: [1 week? 1 month? 6 months?] |
| 24 | |
| 25 | 4. RISK ASSESSMENT |
| 26 | - Main technical risk: [what could make this fail?] |
| 27 | - What would we do if it fails? [is there a fallback direction?] |
| 28 | - Is this already published? [arXiv search, citation check] |
| 29 | - Who else is likely working on this? [are we too late?] |
| 30 | |
| 31 | 5. RESEARCH AGENDA |
| 32 | - If this works: what is the natural follow-on paper? |
| 33 | - What related problems does this technique apply to? |
| 34 | - What is the 3-year research direction this could seed? |
| 35 | """ |
| 36 | |
| 37 | # โโโ Example: Evaluated Research Idea โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 38 | |
| 39 | EXAMPLE_IDEA = """ |
| 40 | IDEA: Consistency regularization for LLM factuality |
| 41 | |
| 42 | PROBLEM: |
| 43 | LLMs generate inconsistent facts โ ask the same factual question |
| 44 | differently phrased and you get different answers. This suggests |
| 45 | the model's factual knowledge is shallow and context-dependent. |
| 46 | |
| 47 | HYPOTHESIS: |
| 48 | IF we add an auxiliary loss penalizing inconsistent answers to |
| 49 | semantically equivalent questions during RLHF, |
| 50 | THEN TruthfulQA accuracy will improve by 15-25% |
| 51 | BECAUSE forcing internal consistency requires the model to have |
| 52 | more robust factual representations (not just surface pattern matching) |
| 53 | |
| 54 | NULL HYPOTHESIS: |
| 55 | Inconsistency is due to decoding stochasticity, not knowledge |
| 56 | โ temperature reduction achieves the same benefit without training. |
| 57 | |
| 58 | MINIMAL EXPERIMENT: |
| 59 | - Benchmark: TruthfulQA (readily available), FactScore |
| 60 | - Baseline: standard RLHF (InstructGPT recipe), temperature=0 |
| 61 | - Method: generate N paraphrases per question, add consistency |
| 62 | loss = MSE(logits_1, logits_2) to the RLHF reward function |
| 63 | - Success: +10% TruthfulQA accuracy without degrading MMLU |
| 64 | - Time estimate: 2 weeks (use a 7B model for fast iteration) |
| 65 | |
| 66 | RISKS: |
| 67 | - Main risk: consistency doesn't generalize โ model learns |
| 68 | consistent but uniformly wrong answers |
| 69 | - Fallback: use consistency as a filtering signal for RLHF |
| 70 | training data rather than an explicit loss |
| 71 | - Prior work: Consistency in LLMs (Elazar et al. 2021) studied |
| 72 | probing, not training. No direct prior work on this loss. |
| 73 | - Competition: likely being tried at major labs โ need to |
| 74 | execute quickly and check arXiv weekly |
| 75 | |
| 76 | AGENDA: |
| 77 | Paper 1: consistency regularization for factuality |
| 78 | Paper 2: extend to multi-hop factual reasoning chains |
| 79 | Paper 3: consistency as a signal for retrieval augmentation |
| 80 | """ |
| 81 | |
| 82 | print(IDEA_EVALUATION_TEMPLATE) |
| 83 | print("=" * 70) |
| 84 | print("EXAMPLE APPLICATION:") |
| 85 | print(EXAMPLE_IDEA) |
| 86 | |
| 87 | # โโโ Open problems by area (for interview preparation) โโโโโโโโโโโโโโโโโโโโโโ |
| 88 | OPEN_PROBLEMS = { |
| 89 | "LLMs / Alignment": [ |
| 90 | "Scalable oversight: how to train alignment for tasks humans can't verify", |
| 91 | "Deceptive alignment detection: how to test if a model is 'playing along'", |
| 92 | "Factuality vs. fluency tradeoff: current RLHF improves one at expense of other", |
| 93 | "Systematic reasoning: LLMs still fail at multi-step arithmetic and logic", |
| 94 | "Out-of-distribution generalization: models fail on distributions far from training", |
| 95 | ], |
| 96 | "Efficiency": [ |
| 97 | "Compute-optimal inference (vs training): what architectures minimize inference FLOP for given capability?", |
| 98 | "Long-context efficiency: O(nยฒ) attention is a barrier for 1M+ context", |
| 99 | "Mixture of Experts at inference: routing instability and load balancing", |
| 100 | "Quantization with no quality loss: INT4 still degrades on reasoning tasks", |
| 101 | ], |
| 102 | "Multimodal": [ |
| 103 | "Grounding: VLMs still confuse spatial relationships in images", |
| 104 | "Compositional reasoning: 'the red cube on top of the blue sphere' fails often", |
| 105 | "Video understanding at scale: temporal reasoning over long videos", |
| 106 | "Audio+video alignment: synchronizing speech, sound, and visual events", |
| 107 | ], |
| 108 | } |
| 109 | |
| 110 | for area, problems in OPEN_PROBLEMS.items(): |
| 111 | print(f"\n{area}:") |
| 112 | for p in problems: |
| 113 | print(f" โข {p}") |
A structured research idea evaluation framework with a concrete example: consistency regularization for LLM factuality. Each component shows what a mature Research Scientist considers โ not just 'will this work?' but 'what's the minimal experiment, what's the null hypothesis, and what's the 3-paper research agenda?' The open problems list shows awareness of where the field is stuck.
Worked Interview Problems
3 problemsProblem
You're interviewing for a Research Scientist role at an AI lab working on LLM alignment. How do you answer this question to demonstrate research vision?
Solution
Read the team's 5 most recent papers before the interview. Understand their current research directions. Your answer should connect to something they're already working on, extend it, or address a clear gap.
Frame around an open problem, not a technique: 'I'm most interested in the scalable oversight problem โ as models become more capable than humans at certain tasks, how do we train alignment for those tasks? Your recent work on [paper X] addresses this by [approach Y]. I see a natural extension: [your idea].'
Propose a specific research agenda with a minimal first experiment: 'My first 6 months would focus on testing whether debate-based oversight (two models arguing, human judges) generalizes beyond toy domains. The minimal experiment: take the debate setup from Irving et al. (2018), extend to a domain where humans can evaluate correctness (factual QA), and measure whether the honest debater wins more often than in adversarial settings. If yes, this suggests debate scales beyond the theoretical regime where it was originally validated.'
Show risk awareness: 'The main risk is that humans aren't good enough judges of debate quality on hard topics โ that's actually the key question I'd want to answer. If that's true, it suggests we need to focus on human training (can we make humans better judges?) rather than AI debate technique.'
Connect to the team's work: 'I'd want to build on your paper [X] by using your [method] as the debate framework and extending it to [domain]. I think the key insight from your work โ [specific insight] โ suggests this direction is viable because [specific reason].'
Answer
Structure: identify open problem โ propose specific research agenda โ minimal first experiment โ risk assessment โ connection to team's work. The specificity (named papers, named datasets, concrete experimental setup) distinguishes this answer from vague 'I'd work on alignment' answers.
Common Mistakes
Proposing a research idea without knowing if it's already been published. Always do a 5-minute arXiv search before discussing a 'novel' idea in an interview. Proposing something published 6 months ago suggests you're not current with the literature.
Proposing ideas that are too vague: 'I want to improve the reasoning of LLMs.' This tells the interviewer nothing. What specific failure mode? What specific intervention? What specific measurement? Vague ideas suggest vague thinking.
Not knowing the team's recent work before proposing a research agenda. 'I'd work on X' โ when X is exactly what the team published last month โ either shows you haven't done your homework or that you'd duplicate effort. Read the team's papers before the interview.
Proposing only safe incremental ideas. Research Scientist interviews favor ambitious ideas with clear justification. 'I'd slightly improve LoRA by 2%' is a safe answer that doesn't distinguish you. 'I think the fundamental bottleneck in LLM factuality is X, and here's a minimal experiment to test this hypothesis' is ambitious and specific.
Not having an answer to 'what's the main risk of your idea?' Every research idea can fail. Knowing why yours might fail โ and having a fallback direction โ signals scientific maturity. Not knowing the risks signals overconfidence.