Canonical: https://knowlpedia.com/probability/shannon-entropy/

# Shannon entropy

A measure of uncertainty of a discrete random variable, defined from its probability mass function.

The **Shannon entropy** of a discrete [random variable](https://knowlpedia.com/probability/random-variable/) \(X\) with [probability mass function](https://knowlpedia.com/probability/probability-mass-function/) \(p(x)=\mathbb{P}(X=x)\) is
\[
H(X) \;=\; -\sum_x p(x)\,\log p(x),
\]
with the convention \(0\log 0=0\). The value may be \(+\infty\). Unless stated otherwise, \(\log\) denotes the natural logarithm; changing its base rescales \(H\) by a constant factor.

## Equivalent characterizations

Equivalently, \(H(X)\) is the [expectation](https://knowlpedia.com/probability/expectation/) of \(-\log p(X)\) under the distribution of \(X\).

## Remarks

Shannon entropy is closely related to [relative entropy (KL divergence)](https://knowlpedia.com/probability/relative-entropy-kl-divergence/) and is a central quantity in information theory.

## Examples

- If \(X\sim\mathrm{Bernoulli}(p)\), then \(H(X)=-p\log p-(1-p)\log(1-p)\).
- If \(X\) is uniform on \(\{1,2,3,4,5,6\}\), then \(H(X)=\log 6\).
