What are Type I and Type II errors, and how do they trade off?
A Type I error is rejecting a true null hypothesis (false positive), controlled by the significance level alpha. A Type II error is failing to reject a false null hypothesis (false negative), with probability beta. Lowering alpha reduces Type I errors but increases Type II errors, so the right balance depends on the cost of each mistake.
How to think about it
These two error types sit at the heart of any decision-theoretic view of hypothesis testing. Knowing the names is not enough — you need to know the consequences and the levers that control each.
Definitions
| H0 is true | H0 is false | |
|---|---|---|
| Reject H0 | Type I error (alpha) | Correct (power = 1 - beta) |
| Fail to reject H0 | Correct (1 - alpha) | Type II error (beta) |
Type I error (false positive): You conclude there is an effect when there is none. Rate controlled by alpha, the significance level. Setting alpha = 0.05 means you accept a 5% chance of this mistake when H0 is true.
Type II error (false negative): You miss a real effect. Rate is beta; statistical power is 1 - beta. A power of 0.80 means an 80% chance of detecting the effect if it truly exists.
The trade-off visualized
Practical consequences
- Medical diagnostics: A false negative (Type II) on cancer is often more costly than a false positive, so alpha is set higher.
- Spam filtering: A false positive (Type I) deletes a legitimate email; many users prefer stricter thresholds.
- A/B testing at scale: With many simultaneous tests, uncorrected Type I errors accumulate rapidly.
How to reduce each
- Reduce Type I: lower alpha.
- Reduce Type II: increase sample size, increase effect size, or increase alpha (the trade-off).