Principle of maximum entropy

Remarks

Proof idea: for $p \in \mathcal{P}$, a direct computation with the constraints gives $H(p^*) - H(p) = D(p \,\|\, p^*)$. Here $D$ is the Kullback–Leibler divergence, which is nonnegative and vanishes only for $p = p^*$. Example: with one constraint, the mean energy $\sum_\omega p(\omega) E(\omega) = U$, the maximizer is the Boltzmann distribution $e^{-\beta E} / Z$, with $\beta$ chosen to match $U$. With no constraint at all, the maximizer is the uniform distribution, with entropy $\ln \abs{\Omega}$. The existence hypothesis matters: if $f_1 = E$ and $c_1 = \min E$ with a unique minimizer, no exponential family member with finite $\lambda$ matches, and the maximizer is a point mass.

Going deeper

For a student

As a theorem, it says that exponential families are the entropy maximizers under linear constraints. As a principle of inference, proposed by E. T. Jaynes in 1957, it says that one should use these maximizers as the probability assignments. In statistical mechanics it rederives the canonical ensemble from a mean-energy constraint and the grand canonical ensemble from mean energy and mean particle number. The multipliers $\lambda_k$ are found from $c_k = -\partial \ln Z / \partial \lambda_k$, and they play the role of inverse temperature and of $-\beta \mu$.