Canonical: https://knowlpedia.com/probability/maximum-entropy-principle/

# Maximum entropy principle

A rule for selecting a probability distribution by maximizing entropy subject to known constraints.

A **maximum entropy principle** selects, when a maximizer exists, a [probability distribution](https://knowlpedia.com/probability/distribution-law/) \(P^\star\) from a nonempty feasible class \(\mathcal C\) such that
\[
P^\star \in \operatorname*{arg\,max}_{P\in\mathcal C} H(P),
\]

where \(H\) is the chosen entropy functional and \(\mathcal C\) is the feasible class of distributions.

The guiding idea is to choose a distribution that adds as little structure as the entropy model permits beyond the stated constraints. This depends on the chosen entropy and, in the continuous case, on the underlying coordinates or [reference measure](https://knowlpedia.com/measure-theory/measure/). Relative-entropy minimization against a specified reference distribution is the corresponding reference-dependent formulation.

## Examples

For discrete laws, \(H\) is commonly [Shannon entropy](https://knowlpedia.com/probability/shannon-entropy/); for absolutely continuous laws it is often [differential entropy](https://knowlpedia.com/probability/differential-entropy/). A typical feasible class is specified by support or [moment](https://knowlpedia.com/probability/moment/) constraints, for example \(\mathbb E_P[g_i(X)]=c_i\).
- On a finite set of \(n\) outcomes, the [uniform distribution](https://knowlpedia.com/probability/uniform-distribution/) maximizes Shannon entropy.
- Among probability distributions on \(\mathbb R\) with a fixed mean and a fixed positive [variance](https://knowlpedia.com/probability/variance/), the [normal distribution](https://knowlpedia.com/probability/normal-distribution/) with those parameters maximizes differential entropy.
