Quant Research Guide
Backtesting & Evaluation
A backtest is a hypothesis, not a result. The ability to build honest backtests โ free of look-ahead bias, survivorship bias, and overfitting โ and then correctly interpret performance metrics is what separates credible quant researchers from those who fool themselves. Interviewers probe this relentlessly.
Core Theory
- 1Look-ahead bias is the use of information at time t that was not available until after t. It inflates backtest performance and is the #1 source of false alpha. Three forms: (1) signal computed using the current-period return, (2) fundamental data used before filing date, (3) universe of stocks defined using future membership (e.g., current S&P 500 constituents only).
- 2Survivorship bias: backtesting only on stocks that survived to today inflates returns because delisted, bankrupt, and acquired stocks are excluded. Survivorship bias inflates equity factor returns by approximately 1-2% per year (Elton et al. 1996). Always use a point-in-time universe that includes all stocks that existed at each historical date.
- 3Point-in-time (PIT) data: fundamental data (earnings, book value, revenue) is revised after initial announcement. The announced value on date t is different from the finally-restated value. Using final restated data introduces look-ahead. PIT databases (Compustat Point-in-Time, FactSet) store the value as it was known on each date.
- 4Walk-forward validation: train on months 1-36, test on months 37-48, then train on 1-48 test on 49-60, etc. The test period is always strictly after the training period โ no overlap. This is the correct validation methodology for time-series signals and prevents overfitting to a single test period.
- 5Transaction cost models: (1) fixed bp model: cost = c ร |ฮw|, where c โ 5-30bp one-way for liquid equities. (2) Market impact (Kyle 1985, square-root law): impact โ ฯ ร โ(participation rate) ร ฮป. For large trades, impact dominates fixed costs. Total cost = spread + market impact.
- 6Performance metrics: Sharpe ratio (return/risk), Sortino ratio (penalizes only downside vol), Calmar ratio (return/max drawdown), maximum drawdown (peak-to-trough equity decline), hit rate (fraction of positive days), and average win/loss ratio. Each reveals a different dimension of strategy quality.
- 7Capacity estimation: a strategy's capacity is the maximum AUM before own trades move the market against you. Capacity โ ADV ร (cost threshold / price impact coefficient). A strategy with 200% annual turnover and a 200M/yr. If stocks have $10M ADV, trading 5% of ADV daily is borderline feasible โ above this, market impact destroys alpha.
- 8Overfitting taxonomy: (1) in-sample Sharpe inflated by parameter selection, (2) multiple-testing bias from trying many signals, (3) specification search bias from fitting lookback windows, (4) regime overfitting (calibrated on one macro regime). The deflated Sharpe ratio corrects for (1) and (2). Walk-forward corrects for (3) and (4).
Key Formulas
| 1 | import numpy as np |
| 2 | import pandas as pd |
| 3 | |
| 4 | np.random.seed(42) |
| 5 | T, N = 252, 100 # 1 year daily, 100 stocks |
| 6 | |
| 7 | # --- Simulate daily returns and a signal --- |
| 8 | returns = pd.DataFrame(np.random.randn(T, N) * 0.01, |
| 9 | columns=[f'STK{i:03d}' for i in range(N)]) |
| 10 | signal = pd.DataFrame(np.random.randn(T, N), |
| 11 | columns=returns.columns) |
| 12 | |
| 13 | # --- Correct: shift signal 1 day (no look-ahead) --- |
| 14 | signal_lagged = signal.shift(1) |
| 15 | |
| 16 | # --- Long-Short portfolio: top/bottom quintile each day --- |
| 17 | def ls_positions(sig: pd.DataFrame, long_frac=0.2, short_frac=0.2) -> pd.DataFrame: |
| 18 | ranks = sig.rank(axis=1, pct=True) |
| 19 | long_m = (ranks >= (1 - long_frac)).astype(float) |
| 20 | short_m = (ranks <= short_frac).astype(float) |
| 21 | pos = long_m / long_m.sum(axis=1, keepdims=True) - short_m / short_m.sum(axis=1, keepdims=True) |
| 22 | return pos # dollar-neutral: sum of positions โ 0 each day |
| 23 | |
| 24 | positions = ls_positions(signal_lagged) |
| 25 | |
| 26 | # --- Daily P&L (no transaction costs first) --- |
| 27 | pnl_gross = (positions * returns).sum(axis=1) |
| 28 | |
| 29 | # --- Apply transaction costs: 10bp one-way --- |
| 30 | cost_bps = 0.001 # 10bp = 0.1% |
| 31 | turnover = positions.diff().abs().sum(axis=1) / 2 # one-way turnover |
| 32 | pnl_net = pnl_gross - turnover * cost_bps |
| 33 | |
| 34 | # --- Performance metrics --- |
| 35 | def performance(pnl: pd.Series, label: str): |
| 36 | ann_ret = pnl.mean() * 252 |
| 37 | ann_vol = pnl.std() * np.sqrt(252) |
| 38 | sharpe = ann_ret / ann_vol |
| 39 | cum = (1 + pnl).cumprod() |
| 40 | drawdown = (cum / cum.cummax() - 1) |
| 41 | mdd = drawdown.min() |
| 42 | hit_rate = (pnl > 0).mean() |
| 43 | print(f"--- {label} ---") |
| 44 | print(f" Ann. Return : {ann_ret*100:.2f}%") |
| 45 | print(f" Ann. Vol : {ann_vol*100:.2f}%") |
| 46 | print(f" Sharpe : {sharpe:.3f}") |
| 47 | print(f" Max Drawdown: {mdd*100:.2f}%") |
| 48 | print(f" Hit Rate : {hit_rate*100:.1f}%") |
| 49 | print(f" Calmar : {ann_ret/abs(mdd):.3f}") |
| 50 | |
| 51 | performance(pnl_gross, "Gross (no costs)") |
| 52 | performance(pnl_net, "Net (10bp costs)") |
| 53 | |
| 54 | # --- Walk-forward split --- |
| 55 | train_end = int(T * 0.7) |
| 56 | test_pnl = pnl_net.iloc[train_end:] |
| 57 | print(f"\nOut-of-sample (last 30%) Sharpe: {test_pnl.mean()*252 / (test_pnl.std()*np.sqrt(252)):.3f}") |
| 58 | |
| 59 | # --- Demonstrate look-ahead bias --- |
| 60 | signal_la = signal # NOT shifted โ look-ahead |
| 61 | pos_la = ls_positions(signal_la) |
| 62 | pnl_la = (pos_la * returns).sum(axis=1) |
| 63 | print(f"\nLook-ahead Sharpe (inflated): {pnl_la.mean()*252/(pnl_la.std()*np.sqrt(252)):.3f}") |
| 64 | print(f"Correct Sharpe : {pnl_gross.mean()*252/(pnl_gross.std()*np.sqrt(252)):.3f}") |
A complete vectorized long-short backtest from scratch: signal construction with correct shift(1), equal-dollar-weighted long top quintile and short bottom quintile, transaction cost modeling via realized turnover, and all standard performance metrics. Also explicitly demonstrates how look-ahead bias inflates the Sharpe ratio โ always the first sanity check.
Worked Interview Problems
3 problemsProblem
A researcher backtests a value factor (low P/E = buy) on S&P 500 stocks since 2000. The backtest Sharpe = 1.8. List three specific look-ahead bias sources and how to fix each.
Solution
Source 1: Universe construction. Using the current S&P 500 constituent list means backtesting only on stocks that survived โ delisted, bankrupt, and acquired stocks are excluded. Fix: use point-in-time S&P 500 membership data where available; or use an all-stocks universe with survivorship-free data (CRSP).
Source 2: Fundamental data timing. P/E ratios use earnings that may not have been announced when the signal was computed. Quarterly earnings announced in February for Q4 cannot be used before February. Fix: use a PIT (point-in-time) database and tag each fundamental observation with its announcement date.
Source 3: Price data alignment. Closing price on date t should not be used in the signal on date t โ it is only known after market close. The signal must use yesterday's price at minimum. Fix: signal.shift(1) in pandas, or ensure your signal uses only close prices from t-1.
Answer
Three sources: (1) survivorship bias in universe โ fix with PIT membership data. (2) Fundamental data timing โ fix with announcement-date-tagged PIT fundamentals. (3) Same-day price in signal โ fix with .shift(1). After correcting all three, expect Sharpe to drop significantly.
Common Mistakes
Backtesting only on current index constituents (e.g., current S&P 500). This introduces survivorship bias โ all the stocks that went bankrupt, were delisted, or were acquired are excluded, inflating returns by 1-2%/yr.
Using 'as-reported' fundamental data instead of point-in-time data. Quarterly earnings reported in February for Q4 cannot be used in January. Using the final restated value instead of the initially announced value inflates fundamental signals significantly.
Computing max drawdown from the wrong formula. The correct formula is the maximum peak-to-trough decline in equity value, not the maximum single-day loss. Many candidates confuse these.
Reporting only in-sample performance without walk-forward or out-of-sample validation. Any backtest without an OOS test is not credible. The IS-to-OOS degradation ratio is the key diagnostic.
Ignoring bid-ask spread in cost models. For small/mid-cap stocks, the spread alone can be 20-50bp โ larger than the per-trade impact model. For liquid large-caps the spread is 1-2bp. Use asset-class-appropriate cost assumptions.