Statistical hypothesis test

Summary

A statistical hypothesis test is a rule that uses data to decide whether to reject a hypothesis about the law of the data. One fixes a null hypothesis $H_0$ and an alternative hypothesis $H_1$, for example $H_0 \colon \theta \in \Theta_0$ and $H_1 \colon \theta \in \Theta_1$ with disjoint sets $\Theta_0$ and $\Theta_1$. One computes a test statistic $T(X)$ and rejects $H_0$ when $T(X)$ is in a critical region $C$. The region is chosen so that $P_\theta(T(X) \in C) \le \alpha$ for each $\theta \in \Theta_0$, for a significance level $\alpha$ such as $0.05$.

Remarks

Equivalently, one can compute the p-value and reject $H_0$ when it is at most $\alpha$. A test can make two types of errors: reject a true $H_0$, or fail to reject a false $H_0$. The level $\alpha$ bounds the probability of the first error. The power of the test at $\theta \in \Theta_1$ is $P_\theta(T(X) \in C)$, the probability to reject $H_0$ when it is false. Not rejecting $H_0$ is not a proof that $H_0$ is true.

Explanations

L1 human

A hypothesis test asks whether the data are surprising under a default assumption, the null hypothesis. If the data are too surprising, one rejects the assumption. The significance level fixes how often one accepts to reject a true assumption.