Canonical: https://knowlpedia.com/probability/differential-entropy/

# Differential entropy

The negative integral of a probability density times its logarithm, relative to Lebesgue measure.

Let \(X\) be an \(\mathbb R^d\)-valued [random variable](https://knowlpedia.com/probability/random-variable/) whose law has [density](https://knowlpedia.com/probability/probability-density-function/) \(f\) with respect to [Lebesgue measure](https://knowlpedia.com/measure-theory/lebesgue-measure/). If the integral below is well-defined as an extended integral, the **differential entropy** of \(X\) is
\[
h(X)=-\int_{\mathbb R^d}f(x)\log f(x)\,dx,
\]
with \(0\log0:=0\).

The value may be \(+\infty\) or \(-\infty\). It is undefined if the positive and negative parts of \(f\log f\) both have infinite integral.

## Dependence on coordinates

Differential entropy is entropy relative to a chosen [reference measure](https://knowlpedia.com/measure-theory/measure/), here Lebesgue measure. It is therefore not invariant under changes of coordinates. If \(A\in\mathrm{GL}_d(\mathbb R)\), \(b\in\mathbb R^d\), and the entropies are defined, then
\[
h(AX+b)=h(X)+\log|\det A|.
\]
More generally, for a sufficiently regular [diffeomorphism](https://knowlpedia.com/fiber-bundles/diffeomorphism/) \(g\),
\[
h(g(X))
=
h(X)+\mathbb E\!\left[\log|\det Dg(X)|\right]
\]
whenever both sides are well-defined.

Unlike [Shannon entropy](https://knowlpedia.com/probability/shannon-entropy/), differential entropy can be negative. Coordinate-invariant comparisons are usually expressed through [relative entropy](https://knowlpedia.com/probability/relative-entropy-kl-divergence/).

## Examples

- If \(X\sim\mathcal N(\mu,\Sigma)\) on \(\mathbb R^d\) with positive-definite [covariance](https://knowlpedia.com/probability/covariance/) \(\Sigma\), then
  \[
  h(X)=\frac12\log\!\bigl((2\pi e)^d\det\Sigma\bigr).
  \]
- If \(X\) is uniform on a [measurable set](https://knowlpedia.com/measure-theory/measurable-set/) \(A\subset\mathbb R^d\) with \(0<\lambda(A)<\infty\), then \(h(X)=\log\lambda(A)\).

## References

1. Thomas M. Cover and Joy A. Thomas, *Elements of Information Theory*, 2nd ed., Wiley, 2006, Chapter 8. [Publisher record](https://doi.org/10.1002/047174882X).
