What is the difference between a biased estimator and an inconsistent estimator?
Bias measures the systematic error of an estimator at a fixed sample size — whether its expected value equals the true parameter. Consistency is an asymptotic property — whether the estimator converges in probability to the true parameter as sample size grows to infinity. An estimator can be biased yet consistent, or unbiased yet inconsistent.
How to think about it
Bias and consistency are orthogonal properties. Conflating them is a common interview error that reveals a shallow understanding of estimation theory.
Definitions
Bias: For estimator θ̂ of true parameter θ:
Bias(θ̂) = E[θ̂] - θAn estimator is unbiased if Bias = 0 for all θ and all n. The key word is for all n — it is a finite-sample guarantee.
Consistency: θ̂ₙ is consistent if, for any ϵ > 0:
P(|θ̂ₙ - θ| > ε) → 0 as n → ∞Equivalently, θ̂ₙ converges in probability to θ.
The four combinations
| Unbiased | Biased | |
|---|---|---|
| Consistent | Sample mean for μ | Sample variance s² = ∑(xᵢ - x̄)²/n (divides by n, not n-1) |
| Inconsistent | Sample mean from only first observation (θ̂ = x₁) | Constant estimator θ̂ = 42 |
Biased but consistent — sample variance with divisor n:
σ̂² = (1/n) ∑(xᵢ - x̄)²This underestimates σ² by a factor of (n-1)/n. The bias is -σ²/n, which shrinks to 0 as n → ∞, so it is consistent despite being biased.
Unbiased but inconsistent — using only the first observation:
θ̂ = x₁E[x₁] = μ, so it is unbiased. But it never gets closer to μ as n grows — its variance remains σ² regardless of sample size.
Bias-variance tradeoff connection
Mean Squared Error decomposes as:
MSE(θ̂) = Variance(θ̂) + Bias(θ̂)²A small amount of bias can be worth tolerating if it substantially reduces variance (hence ridge regression). Consistency tells you the MSE eventually collapses; bias tells you whether the collapse is centred on the truth.