Shannon–McMillan–Breiman theorem

Statement

Let $T$ be ergodic and probability preserving, and let $\mathcal P$ be a finite measurable partition. With $\mathcal P^{(n)}=\bigvee_{j=0}^{n-1}T^{-j}\mathcal P$ ,

-\frac1n\log\mu(\mathcal P^{(n)}(x))\longrightarrow h_\mu(T,\mathcal P) \quad\text{for almost every }x.

This is the finite-partition Shannon–McMillan–Breiman theorem. The union of zero-probability cells over all $n$ is a null set and can be omitted.

Interpretation

For a typical trajectory, the probability of observing its length- $n$ name is $\exp(-n h_\mu(T,\mathcal P)+o(n))$ . If the partition is generating, its entropy rate is the entropy of the entire system. The result extends the independent-symbol information law to dependent observations in any ergodic system.

References

Karma Dajani, Introduction to Ergodic Theory and its Applications to Number Theory, 2014, Theorem 5.4.2.