Gibbs' inequality (lemma)

Definitions and notation

• Probability measures , expectation .
• Relative entropy (KL divergence) .
• Shannon entropy (for the cross-entropy interpretation).

Statement

Let $P$ and $Q$ be probability measures on the same measurable space $(\Omega,\mathcal F)$ with $P \ll Q$ (absolute continuity). Let $f=\frac{dP}{dQ}$ be the Radon–Nikodym derivative. Then the relative entropy satisfies

Moreover, $D(P\|Q)=0$ if and only if $P=Q$ (equivalently, $f=1$ $Q$-a.s.).

In the discrete case, for probability vectors $(p_i)_i$ and $(q_i)_i$ with $p_i>0 \Rightarrow q_i>0$,

with equality iff $p_i=q_i$ for all $i$.

Key hypotheses and conclusions

Hypotheses

• $P$ and $Q$ are probability measures on the same space.
• Absolute continuity $P\ll Q$ (so $\frac{dP}{dQ}$ exists).
• Integrability of $\log\!\big(\frac{dP}{dQ}\big)$ under $P$ (automatic in many finite/discrete settings).

Conclusions

• Nonnegativity: $D(P\|Q)\ge 0$.
• Rigidity of equality: $D(P\|Q)=0$ iff the two measures coincide.

Proof idea / significance

A common proof is a one-line application of Jensen's inequality to the convex function $x\mapsto x\log x$ (or to $x\mapsto -\log x$ after normalization). Another elementary route uses the pointwise inequality $\log x \le x-1$, applied to $x=\frac{dQ}{dP}$ on the set where $P$ has mass.

In statistical mechanics, Gibbs’ inequality underlies “maximum entropy” and free-energy variational principles; it is a measure-theoretic form of the statement that the Gibbs distribution minimizes free energy (or equivalently maximizes entropy subject to constraints).