Glivenko–Cantelli theorem

Remarks

For each fixed $x$, the strong law of large numbers applied to the indicators $\mathbf{1}_{\set{X_k \le x}}$ gives $F_n(x) \to F(x)$ almost surely. The same holds for the left limits $F_n(x-) \to F(x-)$. To make the convergence uniform, choose finitely many points $x_0 < x_1 < \cdots < x_m$ such that $F$ grows by at most $\varepsilon$ between consecutive points, apart from jumps at the points themselves. Since $F_n$ and $F$ are non-decreasing, the uniform error is at most $\varepsilon$ plus the error at these finitely many points.

Example: if the $X_k$ are uniform on $[0, 1]$, then the step function $F_n$ lies within any fixed distance of the line $F(x) = x$ on $[0, 1]$, for all large $n$, almost surely.

The rate is given by the Dvoretzky–Kiefer–Wolfowitz inequality, and the fluctuations of order $n^{-1/2}$ are described by the empirical process and Donsker's theorem.

Going deeper

For a student

The theorem is often called the fundamental theorem of statistics, since it justifies estimating a distribution by its empirical version. It is the first uniform law of large numbers. Empirical process theory generalises it to classes of sets or functions, called Glivenko–Cantelli classes, on which the empirical averages converge uniformly; the Vapnik–Chervonenkis condition is a sufficient criterion.