Bias of an estimator

Definition

Let $\hat{\theta}$ be an estimator of a real parameter $\theta$, with $E_\theta[|\hat{\theta}|] < \infty$ for each $\theta$. Here $E_\theta$ is the expectation when $\theta$ is the true parameter. The bias of $\hat{\theta}$ at $\theta$ is $$\operatorname{Bias}_\theta(\hat{\theta}) = E_\theta[\hat{\theta}] - \theta.$$ The estimator is unbiased if $\operatorname{Bias}_\theta(\hat{\theta}) = 0$ for each $\theta$.

Remarks

For an i.i.d. sample with mean $\mu$, the sample mean is an unbiased estimator of $\mu$. The sample variance with divisor $n$ has bias $-\sigma^2/n$. With divisor $n - 1$, it is unbiased. An unbiased estimator need not be good. The mean squared error is variance plus squared bias, and a biased estimator can have a smaller mean squared error. Unbiasedness is not preserved by nonlinear maps. For example, the sample standard deviation $S_n$ is a biased estimator of $\sigma$, even though $S_n^2$ is unbiased for $\sigma^2$.

Explanations

L1 human

An estimator is unbiased if it is right on average, over many repetitions of the experiment. Bias is a systematic error: a scale that always shows $1$ kg too much has a bias of $1$ kg.