Probability Theory

Core idea

Foundations

• Probability measure
• Probability space
• Event

Random Variables

• Random variable
• Random vector
• Distribution (law)
• Identically distributed
• IID sequence

Expectation and Moments

• Expectation
• Expectation of a function
• Conditional expectation
• Conditional probability
• Variance
• Covariance
• Correlation coefficient
• Moment

Generating Functions

• Moment generating function
• Characteristic function
• Cumulant generating function
• Cumulant

Independence

• Independence of events
• Independence of σ-algebras
• Independence of random variables

Concentration Inequalities

• Markov inequality
• Chebyshev inequality
• Chernoff bound

Limit Theorems

• Weak Law of Large Numbers
• Strong Law of Large Numbers
• Central Limit Theorem

Information Theory

• Shannon entropy
• Differential entropy
• Relative entropy (KL divergence)
• Gibbs inequality
• Total variation distance
• Pinsker inequality
• Maximum entropy principle

Measure-Theoretic Foundations

• Radon–Nikodym theorem

Cones, averages and moment constraints

• Covariance matrix
• Derivatives of the log moment generating function
• Exponential tilt
• Variance of a normalized exponential weight

Connections with ergodic theory

• Standard Gaussian probability space
• Probability integral transform

See the ergodic theory reading paths for the surrounding constructions and theorems.