{
  "aliases": null,
  "backlinks": [
    {
      "field": "about",
      "target": "QWEF2WAXJM"
    },
    {
      "field": "uses",
      "target": "QWEF2WAXJM"
    },
    {
      "field": "defines",
      "target": "WW7HPC3WKA"
    }
  ],
  "body": "\n## Summary\n\nThe Kullback–Leibler divergence $D(P \\,\\|\\, Q)$, also called relative entropy, measures how much a probability distribution $P$ differs from a reference distribution $Q$.\nIt is non-negative and equals $0$ exactly when $P = Q$, but it is not symmetric and is not a metric.\nIt is the rate function in [[Sanov's theorem|QWEF2WAXJM]] and the expected log-likelihood ratio in statistics.\nIn information theory, it is the extra code length when one codes for $Q$ but the data follow $P$.\n\n## Explanations\n\n### L1 human\n\nSuppose you believe that data come from a distribution $Q$, but they really come from $P$.\nThe Kullback–Leibler divergence measures, on average, how surprised you are by this mistake.\nFor a coin that really lands heads with probability $1/2$, the divergence from a model that says $1/4$ is $\\frac{1}{2} \\log \\frac{4}{3}$, about $0.14$ in natural units.\nIt is zero only when the model is right.\n",
  "commit": "547b6eddb8f2feb8d579f2d633d22a27b7e132fc",
  "front": {
    "kind": "concept",
    "msc": [
      "94A17",
      "62B10",
      "60F10"
    ],
    "name": "Kullback–Leibler divergence",
    "same_as": {
      "wikidata": "Q255166",
      "wikipedia": "Kullback–Leibler divergence"
    },
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence"
      }
    ],
    "tag": "PQBW43X5RE"
  },
  "kind": "concept",
  "link": "https://math2.tamnd.com/node/PQBW43X5RE/",
  "links": null,
  "markdown": "https://math2.tamnd.com/node/PQBW43X5RE.md",
  "msc": [
    "94A17",
    "62B10",
    "60F10"
  ],
  "name": "Kullback–Leibler divergence",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/PQB/PQBW43X5RE.md",
  "status": null,
  "tag": "PQBW43X5RE",
  "uri": "m2://PQBW43X5RE",
  "url": "https://math2.tamnd.com/wiki/kullback-leibler-divergence/",
  "writing": {
    "tag": "PQBW43X5RE",
    "kind": "concept",
    "level": "W0",
    "next": [
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}