Relative entropy (KL divergence)

A relative entropy (Kullback–Leibler divergence) is an extended real number $D_{\mathrm{KL}}(P\|Q)$ associated to two probability measures $P$ and $Q$ on the same measurable space, defined (when $P$ is absolutely continuous with respect to $Q$) by

where $\frac{dP}{dQ}$ is the Radon–Nikodym derivative (see the Radon–Nikodym theorem ). If $P$ is not absolutely continuous with respect to $Q$, one sets $D_{\mathrm{KL}}(P\|Q)=+\infty$.

In the discrete case with mass functions $p,q$ on a countable set, this becomes

with the convention that terms with $p(x)=0$ contribute $0$, and any $x$ with $p(x)>0$ and $q(x)=0$ forces $D_{\mathrm{KL}}(P\|Q)=+\infty$. Relative entropy is always nonnegative by Gibbs' inequality , equals $0$ iff $P=Q$ (in the appropriate sense), and is not symmetric in general. It is related to other discrepancy notions such as total variation distance (for example via Pinsker's inequality ).

Examples:

• If $P=\mathrm{Bernoulli}(p)$ and $Q=\mathrm{Bernoulli}(q)$ with $p,q\in(0,1)$, then $D_{\mathrm{KL}}(P\|Q)=p\log\frac{p}{q}+(1-p)\log\frac{1-p}{1-q}.$
• If $P=\mathcal{N}(\mu_1,\sigma^2)$ and $Q=\mathcal{N}(\mu_2,\sigma^2)$ with the same variance $\sigma^2>0$, then $D_{\mathrm{KL}}(P\|Q)=\frac{(\mu_1-\mu_2)^2}{2\sigma^2}.$