Total variation distance

A total variation distance is the quantity

for two probability measures $P,Q$ on the same $(\Omega,\mathcal F)$, where the supremum ranges over all measurable sets $A$ (events). It measures the worst-case discrepancy in event probabilities between $P$ and $Q$.

If $P$ and $Q$ are both absolutely continuous with respect to a common measure $\mu$ and have densities $p=\frac{dP}{d\mu}$ and $q=\frac{dQ}{d\mu}$ (via the Radon–Nikodym derivative from Radon–Nikodym theorem ), then

Total variation is a strong notion of closeness of laws , and it can be controlled by KL divergence through Pinsker's inequality .

Examples:

• If $P=\mathrm{Bernoulli}(p)$ and $Q=\mathrm{Bernoulli}(q)$ on $\{0,1\}$, then $d_{\mathrm{TV}}(P,Q)=|p-q|$.
• If $P$ and $Q$ have probability mass functions $(p_i)$ and $(q_i)$ on a finite set, then $d_{\mathrm{TV}}(P,Q)=\frac12\sum_i |p_i-q_i|$.