Kullback–Leibler divergence

Summary

The Kullback–Leibler divergence $D(P \,\|\, Q)$, also called relative entropy, measures how much a probability distribution $P$ differs from a reference distribution $Q$. It is non-negative and equals $0$ exactly when $P = Q$, but it is not symmetric and is not a metric. It is the rate function in Sanov's theorem and the expected log-likelihood ratio in statistics. In information theory, it is the extra code length when one codes for $Q$ but the data follow $P$.