Maximum likelihood estimation

Summary

Maximum likelihood estimation is a method to build estimators of unknown parameters. For observed data $x$, the maximum likelihood estimate is a value $\hat{\theta}$ that maximizes the likelihood function $\theta \mapsto L(\theta \mid x)$ over the parameter set $\Theta$. In other words, it is the parameter value under which the observed data are most probable.

Remarks

The maximizer of $L$ is also the maximizer of the log-likelihood $\ell = \ln L$. For a smooth $\ell$ and an interior maximum, $\hat{\theta}$ solves the likelihood equations $\partial \ell / \partial \theta_j = 0$. Example: for $k$ successes in $n$ Bernoulli($p$) trials, $\ell(p) = k \ln p + (n - k) \ln(1 - p)$, so $\hat{p} = k/n$. Example: for an i.i.d. $\mathcal{N}(\mu, \sigma^2)$ sample, $\hat{\mu} = \bar{x}$ and $\hat{\sigma}^2 = \frac{1}{n}\sum_i (x_i - \bar{x})^2$. This estimator of $\sigma^2$ is biased. Under regularity conditions, the estimator is consistent and asymptotically normal. Also, for a function $g$, the maximum likelihood estimate of $g(\theta)$ is $g(\hat{\theta})$.

Explanations

L1 human

Maximum likelihood chooses the parameter that makes the observed data most probable. If a coin shows $7$ heads in $10$ tosses, the maximum likelihood estimate of the probability of heads is $7/10$.