{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Definition\n\nLet $(\\Omega, \\mathcal{F})$ be a measurable space, and let $P$ and $Q$ be [[probability measures|059TJN0NZ1]] on it.\nIf $P$ is [[absolutely continuous|P0P21KDV6E]] with respect to $Q$, let $dP/dQ$ be the [[Radon–Nikodym derivative|1NTKG64F7N]], and define\n$$D(P \\,\\|\\, Q) = \\int_{\\Omega} \\log \\frac{dP}{dQ} \\, dP \\in [0, \\infty].$$\nIf $P$ is not absolutely continuous with respect to $Q$, define $D(P \\,\\|\\, Q) = \\infty$.\nThe number $D(P \\,\\|\\, Q)$ is the *Kullback–Leibler divergence* of $P$ from $Q$.\n\n## Remarks\n\nThe integral is well defined: with $f = dP/dQ$, it equals $\\int f \\log f \\, dQ$, and $f \\log f \\ge -1/e$, so its negative part is integrable.\nBy Jensen's inequality it is non-negative, with equality exactly when $P = Q$; this is Gibbs' inequality.\n\nFor distributions on a countable set with [[probability mass functions|0D6J4B671D]] $p$ and $q$,\n$$D(P \\,\\|\\, Q) = \\sum_{x} p(x) \\log \\frac{p(x)}{q(x)},$$\nwith the conventions $0 \\log \\frac{0}{q} = 0$ and $p \\log \\frac{p}{0} = \\infty$ for $p \u003e 0$.\nWith densities $p$ and $q$ with respect to a common measure, the sum becomes an integral.\nThe base of the [[logarithm|04DXAFN2ES]] fixes the unit: base $e$ gives nats, base $2$ gives bits.\n\nExample: let $P$ be the Bernoulli law with parameter $1/2$ and $Q$ the Bernoulli law with parameter $1/4$.\nThen $D(P \\,\\|\\, Q) = \\frac{1}{2} \\log \\frac{1/2}{1/4} + \\frac{1}{2} \\log \\frac{1/2}{3/4} = \\frac{1}{2} \\log \\frac{4}{3}$.\nIn the other order, $D(Q \\,\\|\\, P) = \\frac{1}{4} \\log \\frac{1}{2} + \\frac{3}{4} \\log \\frac{3}{2} = \\frac{1}{4} \\log \\frac{27}{16}$.\nThese differ, so $D$ is not symmetric and is not a metric.\n\nExample of the infinite case: let $Q$ be the Dirac measure at $0$ and $P$ the Bernoulli law with parameter $1/2$.\nThen $P(\\set{1}) \u003e 0 = Q(\\set{1})$, so $P$ is not absolutely continuous with respect to $Q$, and $D(P \\,\\|\\, Q) = \\infty$.\n\nPinsker's inequality bounds the total variation distance by $\\sqrt{D(P \\,\\|\\, Q)/2}$.\n",
  "commit": "5fc425f97f1f6b11da5892c2948ef8c5d8805665",
  "front": {
    "defines": "PQBW43X5RE",
    "kind": "definition",
    "msc": [
      "94A17",
      "62B10",
      "60F10"
    ],
    "name": "Kullback–Leibler divergence",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence"
      }
    ],
    "tag": "WW7HPC3WKA",
    "uses": [
      "059TJN0NZ1",
      "04DXAFN2ES",
      "0D6J4B671D"
    ]
  },
  "kind": "definition",
  "link": "https://math2.tamnd.com/node/WW7HPC3WKA/",
  "links": [
    {
      "field": "defines",
      "target": "PQBW43X5RE"
    },
    {
      "field": "uses",
      "target": "059TJN0NZ1"
    },
    {
      "field": "uses",
      "target": "04DXAFN2ES"
    },
    {
      "field": "uses",
      "target": "0D6J4B671D"
    }
  ],
  "markdown": "https://math2.tamnd.com/node/WW7HPC3WKA.md",
  "msc": [
    "94A17",
    "62B10",
    "60F10"
  ],
  "name": "Kullback–Leibler divergence",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/WW7/WW7HPC3WKA.md",
  "status": null,
  "tag": "WW7HPC3WKA",
  "uri": "m2://WW7HPC3WKA",
  "url": "https://math2.tamnd.com/wiki/kullback-leibler-divergence-definition/",
  "writing": {
    "tag": "WW7HPC3WKA",
    "kind": "definition",
    "level": "W0",
    "next": [
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}