Canonical: https://knowlpedia.com/probability/gibbs-inequality-kl/

# Gibbs' inequality (nonnegativity of KL divergence)

The Kullback–Leibler divergence is always nonnegative, and it is zero only when the two distributions are identical.

**Gibbs' inequality:** Let \(P\) and \(Q\) be [probability measures](https://knowlpedia.com/probability/probability-measure/) on a [set](https://knowlpedia.com/shared-foundations/set/) \(\Omega\) equipped with a [sigma-algebra](https://knowlpedia.com/measure-theory/sigma-algebra/) \(\mathcal F\), and let \(D(P\|Q)\) denote their [Kullback–Leibler divergence](https://knowlpedia.com/probability/relative-entropy-kl-divergence/) (allowing the value \(+\infty\)). Then
\[
D(P\|Q)\ge 0,
\]

with equality if and only if \(P=Q\) (as measures on \((\Omega,\mathcal F)\)).

## Equivalent characterizations

Equivalently, in the case \(P\ll Q\), writing \(f=\frac{dP}{dQ}\) for the [Radon–Nikodym derivative](https://knowlpedia.com/measure-theory/radon-nikodym-derivative/) (see [Radon–Nikodym theorem](https://knowlpedia.com/probability/radon-nikodym-theorem/)), one has
\[
D(P\|Q)=\int_\Omega f\log f\,dQ \ge 0,
\]

and equality holds if and only if \(f=1\) \(Q\)-almost everywhere.

## Remarks

This is the basic reason relative entropy is a divergence: it is minimized uniquely at the matching law, even though it is not a metric. It is also a key input for inequalities relating KL to [total variation distance](https://knowlpedia.com/probability/total-variation-distance/), such as [Pinsker's inequality](https://knowlpedia.com/probability/pinsker-inequality/).
