Probability Theory

Foundations

• Probability measure
• Probability space
• Event

Random Variables

• Random variable
• Random vector
• Distribution (law)
• Identically distributed
• IID sequence

Expectation and Moments

• Expectation
• Expectation of a function
• Conditional expectation
• Conditional probability
• Variance
• Covariance
• Correlation coefficient
• Moment

Generating Functions

• Moment generating function
• Characteristic function
• Cumulant generating function
• Cumulant

Independence

• Independence of events
• Independence of σ-algebras
• Independence of random variables

Concentration Inequalities

• Markov inequality
• Chebyshev inequality
• Chernoff bound

Limit Theorems

• Weak Law of Large Numbers
• Strong Law of Large Numbers
• Central Limit Theorem

Information Theory

• Shannon entropy
• Differential entropy
• Relative entropy (KL divergence)
• Gibbs inequality
• Total variation distance
• Pinsker inequality
• Maximum entropy principle

Measure-Theoretic Foundations

• Radon–Nikodym theorem