{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Statement\n\nLet $\\Sigma$ be a [[Polish space|584GXXY73M]], and let $M_1(\\Sigma)$ be the set of [[probability measures|059TJN0NZ1]] on $\\Sigma$ with the topology of [[weak convergence|MBW9JCGZC1]].\nLet $X_1, X_2, \\dots$ be [[i.i.d.|00MRN0FARV]] random elements of $\\Sigma$ with law $\\mu$.\nLet $L_n = \\frac{1}{n} \\sum_{k=1}^{n} \\delta_{X_k}$ be the empirical measure, where $\\delta_x$ is the [[Dirac measure|0ACRGYNHSR]] at $x$.\nThen the laws of $L_n$ satisfy the [[large deviation principle|MNPWBBNCAD]] on $M_1(\\Sigma)$ with the good rate function $\\nu \\mapsto D(\\nu \\,\\|\\, \\mu)$, the [[Kullback–Leibler divergence|PQBW43X5RE]] of $\\nu$ from $\\mu$.\n\n## Remarks\n\nFor a finite set $\\Sigma = \\set{1, \\dots, m}$, the proof is a counting argument.\nThe probability that $L_n$ equals a given type $\\nu$, with $n \\nu(i)$ an integer for each $i$, is a multinomial probability.\nBy Stirling's formula, it equals $e^{-n D(\\nu \\| \\mu) + O(\\log n)}$, and there are only polynomially many types.\nThe general case reduces to finite partitions of $\\Sigma$, or uses Cramér's theorem in a space of measures.\n\nExample: for fair coin tosses, $\\mu$ is uniform on $\\set{0, 1}$.\nThe event that the fraction of ones is at least $0.6$ has probability $e^{-n c + o(n)}$, where $c = D(\\nu \\| \\mu)$ for $\\nu = (0.4, 0.6)$, that is $c = 0.6 \\log 1.2 + 0.4 \\log 0.8 \\approx 0.020$.\n\n## Explanations\n\n### L1 human\n\nDraw many independent samples from a known distribution and make a histogram.\nThe histogram almost always resembles the true distribution.\nSanov's theorem says that the chance that it resembles some other distribution $\\nu$ instead is about $e^{-n D(\\nu \\| \\mu)}$, where $D$ is the Kullback–Leibler divergence.\n\n### L2 human\n\nSanov's theorem gives the Kullback–Leibler divergence a probabilistic meaning and is a basic link between probability and information theory.\nBy the contraction principle, it implies Cramér's theorem for bounded continuous functions of the samples.\nIn statistics it yields error exponents of tests, such as Stein's lemma for hypothesis testing.\nThe conditional limit theorem, or Gibbs conditioning principle, also follows: given a rare constraint on $L_n$, the empirical measure concentrates near the minimiser of $D(\\cdot \\| \\mu)$ under the constraint.\n",
  "commit": "547b6eddb8f2feb8d579f2d633d22a27b7e132fc",
  "front": {
    "about": [
      "Q702WCDQQ3",
      "PQBW43X5RE"
    ],
    "kind": "statement",
    "msc": [
      "60F10",
      "94A17"
    ],
    "name": "Sanov's theorem",
    "role": "theorem",
    "same_as": {
      "wikidata": "Q7418899",
      "wikipedia": "Sanov's theorem"
    },
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Sanov%27s_theorem"
      }
    ],
    "tag": "QWEF2WAXJM",
    "uses": [
      "00MRN0FARV",
      "059TJN0NZ1",
      "0ACRGYNHSR",
      "MNPWBBNCAD",
      "PQBW43X5RE"
    ]
  },
  "kind": "statement",
  "link": "https://math2.tamnd.com/node/QWEF2WAXJM/",
  "links": [
    {
      "field": "about",
      "target": "Q702WCDQQ3"
    },
    {
      "field": "about",
      "target": "PQBW43X5RE"
    },
    {
      "field": "uses",
      "target": "00MRN0FARV"
    },
    {
      "field": "uses",
      "target": "059TJN0NZ1"
    },
    {
      "field": "uses",
      "target": "0ACRGYNHSR"
    },
    {
      "field": "uses",
      "target": "MNPWBBNCAD"
    },
    {
      "field": "uses",
      "target": "PQBW43X5RE"
    }
  ],
  "markdown": "https://math2.tamnd.com/node/QWEF2WAXJM.md",
  "msc": [
    "60F10",
    "94A17"
  ],
  "name": "Sanov's theorem",
  "prov": null,
  "role": "theorem",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/QWE/QWEF2WAXJM.md",
  "status": null,
  "tag": "QWEF2WAXJM",
  "uri": "m2://QWEF2WAXJM",
  "url": "https://math2.tamnd.com/wiki/sanovs-theorem/",
  "writing": {
    "tag": "QWEF2WAXJM",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}