Information geometry

Definition

Let $(X, \mu)$ be a measure space and let $\Theta \subseteq \R^n$ be open. A statistical model is a family $\set{p(\cdot\,; \theta) : \theta \in \Theta}$ of positive probability densities on $X$ with respect to $\mu$ such that:

1. $\theta \mapsto p(x; \theta)$ is smooth for every $x \in X$, and derivatives in $\theta$ may be taken under the integral sign;
2. with $\ell(x; \theta) = \log p(x; \theta)$ and $\partial_i = \partial/\partial\theta^i$, the functions $\partial_1 \ell, \dots, \partial_n \ell$ of $x$ are linearly independent for each $\theta$.

The Fisher information metric is the Riemannian metric on $\Theta$ given by $$g_{ij}(\theta) = \mathbb{E}_\theta\left[\partial_i \ell \; \partial_j \ell\right],$$ where $\mathbb{E}_\theta$ is the expected value for the density $p(\cdot\,; \theta)$. For a real number $\alpha$, the $\alpha$-connection is the affine connection on $\Theta$ with $$\Gamma^{(\alpha)}_{ij,k}(\theta) = \mathbb{E}_\theta\left[\left(\partial_i \partial_j \ell + \frac{1 - \alpha}{2}\, \partial_i \ell \; \partial_j \ell\right) \partial_k \ell\right],$$ where $\Gamma^{(\alpha)}_{ij,k} = g\left(\nabla^{(\alpha)}_{\partial_i} \partial_j, \partial_k\right)$. Information geometry is the study of a statistical model through its Fisher metric and its $\alpha$-connections.