Sanov's theorem

Large deviations for empirical measures of an independent identically distributed sample.

Sanov’s theorem

Sanov’s theorem: Let $(X_i)_{i\ge 1}$ be an i.i.d. sequence taking values in a Polish space $E$ , with common law $\mu$ . Define the empirical measure

L_n=\frac{1}{n}\sum_{i=1}^n \delta_{X_i},

viewed as a random element of $\mathcal{P}(E)$ (Borel probability measures on $E$ ) equipped with the topology of weak convergence. Then $(L_n)$ satisfies a large deviation principle on $\mathcal{P}(E)$ with speed $n$ and good rate function

I(\nu)=H(\nu\|\mu)= \begin{cases} \displaystyle \int_E \log\!\left(\frac{d\nu}{d\mu}\right)\,d\nu, & \nu\ll \mu,\\[6pt] +\infty, & \text{otherwise.} \end{cases}

Here $H(\nu\|\mu)$ is relative entropy (Kullback–Leibler divergence) . Combined with the contraction principle , Sanov’s theorem yields many LDPs for functionals of empirical measures.