Markov chain (discrete time)

A discrete-time Markov chain is a basic model for stochastic dynamics and is frequently used for equilibrium sampling (e.g. MCMC) and for coarse-grained dynamics in statistical mechanics.

Prerequisites: probability measure , expectation .

Definition (Markov property)

Let $(X_n)_{n\ge 0}$ be a process on a state space $S$. It is a Markov chain if for all $n$ and measurable $A\subseteq S$,

The conditional law is determined by a transition kernel $P(x,A)$ (or by a matrix $P_{ij}$ when $S$ is countable).

Transition operator and $n$-step transitions

For bounded functions $f$ on $S$ define

and define $n$-step kernels by composition: $P^{n+m}=P^n P^m$ (Chapman–Kolmogorov).

If $\mu_0$ is the initial distribution, then the distribution after $n$ steps is

Stationary distribution

A probability measure $\pi$ on $S$ is stationary if

In statistical mechanics, $\pi$ is often chosen to be an equilibrium distribution such as a canonical ensemble measure.

Detailed balance and reversibility

If $\pi$ is stationary, the chain is reversible with respect to $\pi$ when it satisfies detailed balance :

(in the discrete state case: $\pi_i P_{ij}=\pi_j P_{ji}$).

Expectations along the chain

For an observable $f$ and initial law $\mu_0$,

Time averages and convergence to equilibrium depend on additional properties (irreducibility, aperiodicity, recurrence), but the operator viewpoint above is the main structural tool.