Hypothesis Testing & p-values

Medium3–4 hrsHypothesis Testingp-valueType I/II ErrorsPower

Overview

Hypothesis testing is the formal way to ask 'is this result real or just luck?' The p-value is the probability of seeing results this extreme if there were no effect — it's NOT the probability of the null being true. The biggest practical mistake in quant: testing many signals and reporting only the significant ones. If you test 100 strategies, 5 will be significant at α=5% just by chance. This is why quant firms require t > 3.

How to Recognize

→'Is this strategy significantly profitable?'
→Determine if a parameter differs from a hypothesized value
→Research interviews: 'How would you test if a signal is real?'
→'Are these two populations different?'

Step-by-Step Approach

1.State H₀ (null) and H₁ (alternative) before seeing data
2.Compute test statistic: Z = (X̄ − μ₀)/(σ/√n) or t = (X̄ − μ₀)/(s/√n)
3.p-value = P(test stat this extreme | H₀ true). NOT P(H₀ is true).
4.Reject H₀ if p < α. For quant signals: use α=0.003 (t>3.0)

Key Formulas

Z = \frac{\bar{X}-\mu_0}{\sigma/\sqrt{n}}

, reject if

|Z|>1.96

\alpha=5\%

|Z|>3

\alpha=0.3\%

02Type I error

\alpha

: P(reject H₀ | H₀ true) — false positive

03Type II error

\beta

: P(fail to reject | H₁ true) — false negative

04Power

= 1-\beta = P(\text{reject} | H_1)

Worked Examples

0Quick Example

Strategy: 252 daily returns, mean=0.05%/day, std=1%/day. t = 0.05/(1/√252) = 0.05/0.063 = 0.79. NOT significant (need t>3 for quant).

Problem

You ran a strategy for 100 days. Mean daily P&L = $200, std =$ 1500. Is the strategy profitable at a 5% significance level?

Solution

H₀: μ = 0 (no edge). H₁: μ > 0 (positive edge). One-sided test.

t = (200 − 0)/(1500/√100) = 200/150 = 1.33.

For one-sided 5% test: critical value = 1.645 (Φ(1.645)=0.95).

Our t = 1.33 < 1.645. Fail to reject H₀.

p-value = P(Z > 1.33) = 1 − Φ(1.33) ≈ 1 − 0.908 = 9.2% > 5%.

Not significant at 5% level. We don't have enough evidence of edge — or need more data.

Answer

$t = 1.33 < 1.645$ : not significant at 5% level. Need roughly 140 days with the same parameters to reach t=1.96, and 500+ days for t=3.0.

Common Mistakes

!
p-value of 0.03 does NOT mean P(H₀ is true) = 3%. It means P(data this extreme | H₀ is true) = 3%.
!
Accepting H₀ when you fail to reject. Correct: 'insufficient evidence to reject H₀', not 'H₀ is true'.
!
Choosing α after seeing the data ('fishing'). α must be chosen before the test.

Practice Problems

Click "Show Answer" to reveal

A coin is flipped 1000 times, landing heads 530 times. Test H₀: p=0.5 at α=5%.

Hint: Z = (530/1000 − 0.5) / √(0.5·0.5/1000).

What Type I and Type II errors mean in a medical context: testing if a new drug works.

Hint: H₀: drug has no effect. H₁: drug works.

Back to Study Plan

Poisson & Exponential Nash Equilibrium & Dominant Strategies