Stochastic Convergence and Limit Theorems: Laws of Large Numbers and Central Limit Theorems

In classical deterministic real analysis, the convergence of a sequence of real numbers $\{x_n\}_{n=1}^\infty$ to a fixed limit $x \in \mathbb{R}$ is governed by Cauchy's familiar criterion: for every positive error threshold $\epsilon > 0$, there exists an integer cutoff $N(\epsilon)$ such that for all subsequent indices $n \ge N(\epsilon)$, the Euclidean distance $|x_n - x| < \epsilon$. In mathematical statistics and modern probability theory, however, we deal not with deterministic scalars, but with sequences of random variables $\{X_n\}_{n=1}^\infty$ defined upon an underlying probability space $(\Omega, \mathcal{F}, P)$.

Because random variables are measurable mappings $X_n: \Omega \to \mathbb{R}$ whose numerical realizations depend inherently upon the random outcome $\omega \in \Omega$ selected by nature, the deterministic concept of convergence fractures into four distinct, mathematically profound modes of stochastic convergence. Each mode captures a fundamentally different operational question regarding how, where, and with what regularity a sequence of random quantities settles down as the sample size $n$ approaches infinity.

The Measure-Theoretic Setting

Formally, consider a complete probability space $(\Omega, \mathcal{F}, P)$, where $\Omega$ represents the sample space of all elementary experimental outcomes, $\mathcal{F}$ denotes a $\sigma$-algebra of observable events (subsets of $\Omega$ closed under complements and countable unions), and $P: \mathcal{F} \to [0, 1]$ represents a countably additive probability measure assigning unit mass $P(\Omega) = 1$. A real-valued random variable $X$ is an $(\mathcal{F}, \mathcal{B}(\mathbb{R}))$-measurable function, meaning that for every Borel set $B \in \mathcal{B}(\mathbb{R})$, the pre-image belongs to the event space:

When studying an infinite sequence of random variables $\{X_n\}_{n=1}^\infty$ and a candidate limiting random variable $X$, we must examine the behaviour of the error deviations $|X_n(\omega) - X(\omega)|$. Because the set of sample points $\omega$ where $|X_n(\omega) - X(\omega)|$ is large can vary chaotically from one index $n$ to the next, mathematicians developed four distinct topological frameworks:

1. Almost Sure Convergence ($X_n \xrightarrow{\text{a.s.}} X$)

Almost sure convergence (historically referred to as convergence with probability 1, strong convergence, or convergence almost everywhere) represents the probabilistic equivalent of pointwise convergence in functional analysis. It demands that the set of sample trajectories along which $X_n(\omega)$ fails to converge to $X(\omega)$ as ordinary numbers forms a negligible null set of zero probability.

Formal Definition: A sequence of random variables $\{X_n\}_{n=1}^\infty$ converges almost surely to a random variable $X$, denoted by $X_n \xrightarrow{\text{a.s.}} X$ or $\lim_{n \to \infty} X_n = X \text{ a.s.}$, if:

Equivalently, expressing this requirement in terms of exceptional failure events, the probability that the numerical sequence fails to converge must be identically zero:

Using standard measure-theoretic $\epsilon$-$N$ formulations, almost sure convergence can be expressed as a condition on the supremum of tail deviations. For every arbitrary positive tolerance $\epsilon > 0$:

Alternatively, defining the event of infinite recurrence:

Almost sure convergence requires that $P(|X_n - X| \ge \epsilon \text{ i.o.}) = 0$ for every $\epsilon > 0$. In intuitive terms, almost sure convergence means that if you observe a single realization of the infinite sequence of data, with probability 1 there is some finite, random index $N(\omega, \epsilon)$ after which the entire remaining infinite tail of the sequence remains permanently trapped inside the narrow corridor $(X(\omega) - \epsilon, X(\omega) + \epsilon)$.

2. Convergence in Probability ($X_n \xrightarrow{P} X$)

Convergence in probability (also known as weak convergence of random variables or convergence in measure) relaxes the stringent requirement on infinite trajectories. Rather than demanding that the tail stay trapped forever, it merely asks whether the probability of observing a significant discrepancy at any individual sample index $n$ vanishes as $n$ grows large.

Formal Definition: A sequence of random variables $\{X_n\}_{n=1}^\infty$ converges in probability to a random variable $X$, denoted by $X_n \xrightarrow{P} X$ or $\text{plim}_{n \to \infty} X_n = X$, if for every arbitrary tolerance $\epsilon > 0$:

Equivalently, phrased from the vantage point of proximity:

In the language of statistical estimation, if an estimator sequence $\hat{\theta}_n = T(X_1, \dots, X_n)$ satisfies $\hat{\theta}_n \xrightarrow{P} \theta$ where $\theta$ is the true population parameter, the estimator $\hat{\theta}_n$ is said to be consistent. Consistency is universally regarded as a minimal requirement for any sound statistical procedure: as more data are collected, the probability that the estimate strays from the truth by any noticeable margin collapses to zero.

3. Convergence in Mean of Order $r$ ($X_n \xrightarrow{L^r} X$)

Convergence in mean of order $r$ (or convergence in the Banach space $L^r(\Omega, \mathcal{F}, P)$ for $r > 0$) assesses the distance between random variables by integrating the absolute power difference over the entire probability space.

Formal Definition: Let $r > 0$, and assume that $\mathbb{E}[|X_n|^r] < \infty$ for all $n$ and $\mathbb{E}[|X|^r] < \infty$. The sequence $\{X_n\}$ converges in $r$-th mean to $X$, denoted by $X_n \xrightarrow{L^r} X$, if:

Two special cases are of paramount importance across mathematical statistics:

• When $r = 1$, we obtain convergence in mean (or $L^1$ convergence): $\lim_{n \to \infty} \mathbb{E}[|X_n - X|] = 0$.
• When $r = 2$, we obtain convergence in mean square (or quadratic mean convergence, denoted $X_n \xrightarrow{\text{m.s.}} X$ or $X_n \xrightarrow{L^2} X$):

Mean square convergence is exceptionally useful in applied statistics because the quadratic loss function decomposes naturally into variance and squared bias:

Thus, showing that an estimator's variance and bias both vanish simultaneously is sufficient to establish mean square convergence, which automatically guarantees consistency.

4. Convergence in Distribution ($X_n \xrightarrow{d} X$)

Convergence in distribution (frequently termed convergence in law, or weak convergence of probability measures, denoted by $X_n \xrightarrow{d} X$ or $X_n \xrightarrow{\mathcal{L}} X$) occupies a distinct and special position in asymptotic theory. Unlike the previous three modes, convergence in distribution does not require that $X_n$ and $X$ be defined on the same probability space $\Omega$. It does not track whether $X_n(\omega)$ is numerically close to $X(\omega)$ on individual sample points; instead, it focuses exclusively on the cumulative distribution functions (CDFs) induced by the variables.

Formal Definition: Let $F_n(x) = P(X_n \le x)$ denote the cumulative distribution function of $X_n$, and let $F(x) = P(X \le x)$ denote the cumulative distribution function of $X$. The sequence $\{X_n\}$ converges in distribution to $X$ if:

The restriction to points of continuity of $F(x)$, written $x \in \mathcal{C}(F)$, is mathematically vital. To understand why, consider the deterministic sequence of constant random variables $X_n = 1/n$. Intuitively, $1/n$ converges to the constant $0$. The CDF of $X_n$ is a unit step function located at $1/n$:

The limiting constant variable $X = 0$ has CDF:

Evaluating the pointwise limit of $F_n(x)$ at $x = 0$, we find $F_n(0) = 0$ for all $n \ge 1$, so $\lim_{n \to \infty} F_n(0) = 0 \ne F(0) = 1$. If we demanded convergence at all real numbers, $1/n$ would fail to converge in distribution to $0$! However, $x = 0$ is a jump discontinuity point of $F$. For all $x < 0$, $F_n(x) = 0 \to 0 = F(x)$, and for all $x > 0$, for sufficiently large $n$ such that $1/n < x$, $F_n(x) = 1 \to 1 = F(x)$. Because convergence holds at all points where $F$ is continuous, $X_n \xrightarrow{d} 0$ as expected.

By the Portmanteau Theorem, convergence in distribution is equivalent to several other conditions:

1. $\lim_{n \to \infty} \mathbb{E}[g(X_n)] = \mathbb{E}[g(X)]$ for every bounded, continuous function $g: \mathbb{R} \to \mathbb{R}$.
2. $\limsup_{n \to \infty} P(X_n \in C) \le P(X \in C)$ for all closed sets $C \subset \mathbb{R}$.
3. $\liminf_{n \to \infty} P(X_n \in O) \ge P(X \in O)$ for all open sets $O \subset \mathbb{R}$.
4. $\lim_{n \to \infty} P(X_n \in B) = P(X \in B)$ for all Borel sets $B$ whose boundary has zero limiting probability ($P(X \in \partial B) = 0$).

A central triumph of modern probability theory is the establishment of the precise logical implications connecting these four convergence concepts. The relationships form a strict, unidirectional hierarchical structure:

Fundamental Implications and Their Rigorous Proofs

Theorem 7.1: Almost Sure Convergence Implies Convergence in Probability

If $X_n \xrightarrow{\text{a.s.}} X$, then $X_n \xrightarrow{P} X$.

Proof: Let $\epsilon > 0$ be given. Consider the sequence of events $A_n(\epsilon) = \{|X_n - X| \ge \epsilon\}$. Define the tail union events:

Notice that the sequence of events $\{B_m(\epsilon)\}_{m=1}^\infty$ is nested and monotonically decreasing: $B_{m+1}(\epsilon) \subseteq B_m(\epsilon)$. By the continuity of probability measures from above:

Under the assumption of almost sure convergence, the set of outcomes where $|X_n(\omega) - X(\omega)|$ fails to tend to zero has probability zero. Consequently, the event $\{|X_n - X| \ge \epsilon \text{ i.o.}\}$ has probability zero, meaning:

Now, for any specific index $n$, the single event $A_n(\epsilon)$ is trivially contained within the tail union $B_n(\epsilon)$:

By the monotonicity of probability:

Taking the limit as $n \to \infty$, the upper bound vanishes:

Since probabilities are non-negative, the squeeze theorem forces $\lim_{n \to \infty} P(|X_n - X| \ge \epsilon) = 0$. Because this holds for every $\epsilon > 0$, we conclude that $X_n \xrightarrow{P} X$. $\blacksquare$

Theorem 7.2: $L^r$ Convergence Implies Convergence in Probability

If $X_n \xrightarrow{L^r} X$ for some $r > 0$, then $X_n \xrightarrow{P} X$.

Proof: Fix $\epsilon > 0$. Observe that the event $\{|X_n - X| \ge \epsilon\}$ is identical to the event $\{|X_n - X|^r \ge \epsilon^r\}$. Applying Markov's inequality to the non-negative random variable $|X_n - X|^r$:

By hypothesis, $\lim_{n \to \infty} \mathbb{E}[|X_n - X|^r] = 0$. Since $\epsilon^r > 0$ is a fixed positive constant:

Thus, $X_n \xrightarrow{P} X$. $\blacksquare$

Theorem 7.3: Convergence in Probability Implies Convergence in Distribution

If $X_n \xrightarrow{P} X$, then $X_n \xrightarrow{d} X$.

Proof: Let $F_n(x)$ and $F(x)$ denote the CDFs of $X_n$ and $X$ respectively. Let $x$ be any point of continuity of $F$, and choose an arbitrary $\epsilon > 0$. We decompose the event $\{X_n \le x\}$ based on whether $|X_n - X| < \epsilon$ or $|X_n - X| \ge \epsilon$:

If $X_n \le x$ and $|X_n - X| < \epsilon$, it follows that $X = X_n + (X - X_n) \le x + \epsilon$. Therefore:

Taking probabilities of both sides:

Taking the limit superior as $n \to \infty$, and noting that $P(|X_n - X| \ge \epsilon) \to 0$ by convergence in probability:

Conversely, consider the event $\{X \le x - \epsilon\}$:

Hence:

Subtracting $P(|X_n - X| \ge \epsilon)$ and taking the limit inferior as $n \to \infty$:

Combining our two inequalities:

Because this holds for every $\epsilon > 0$, we take the limit as $\epsilon \to 0^+$. Since $x$ is a point of continuity of $F$, $\lim_{\epsilon \to 0^+} F(x - \epsilon) = F(x)$ and $\lim_{\epsilon \to 0^+} F(x + \epsilon) = F(x)$. By the squeeze theorem:

This establishes that $X_n \xrightarrow{d} X$. $\blacksquare$

Theorem 7.4: Higher-Order $L^s$ Convergence Implies Lower-Order $L^r$ Convergence

Let $0 < r < s$. If $X_n \xrightarrow{L^s} X$, then $X_n \xrightarrow{L^r} X$.

Proof: This is a direct consequence of Lyapunov's inequality (which follows from Jensen's inequality applied to the convex function $g(u) = u^{s/r}$ on $[0, \infty)$):

Raising both sides to the power $r$:

As $n \to \infty$, the right-hand side converges to zero because $\mathbb{E}[|X_n - X|^s] \to 0$ and $r/s > 0$. Thus $\mathbb{E}[|X_n - X|^r] \to 0$. $\blacksquare$

Canonical Counterexamples Showing Non-Equivalence

To grasp why the reverse implications fail in general, we examine three classical counterexamples that every statistician must understand.

Counterexample 1: Convergence in Probability Does NOT Imply Almost Sure Convergence (The "Typewriter Sequence")

Let the sample space be the unit interval $\Omega = [0, 1)$ equipped with the Borel $\sigma$-algebra $\mathcal{F} = \mathcal{B}([0, 1))$ and the standard uniform Lebesgue measure $P = \lambda$, so that for any interval $[a, b) \subseteq [0, 1)$, $P([a, b)) = b - a$.

We construct a sequence of intervals $I_n$ of shrinking widths that continually sweep across the interval $[0, 1)$ like the carriage of a mechanical typewriter:

• Generation 0 ($k = 0$, width 1): $I_1 = [0, 1)$.
• Generation 1 ($k = 1$, width $1/2$): $I_2 = [0, 1/2)$, $I_3 = [1/2, 1)$.
• Generation 2 ($k = 2$, width $1/4$): $I_4 = [0, 1/4)$, $I_5 = [1/4, 2/4)$, $I_6 = [2/4, 3/4)$, $I_7 = [3/4, 1)$.
• In general, for any integer $n \ge 1$, there exists a unique integer $k \ge 0$ and index $j \in \{0, 1, \dots, 2^k - 1\}$ such that $n = 2^k + j$. We define the interval:

Now define the random variable $X_n(\omega) = \mathbf{1}_{I_n}(\omega)$, which takes value 1 if $\omega \in I_n$ and 0 otherwise. Let the candidate limit be the zero random variable $X(\omega) \equiv 0$.

1. Verify Convergence in Probability:

Take any tolerance $\epsilon \in (0, 1)$. The absolute deviation satisfies $|X_n(\omega) - 0| \ge \epsilon$ if and only if $X_n(\omega) = 1$, which occurs precisely when $\omega \in I_n$. Therefore:

As $n \to \infty$, the generation index $k = \lfloor \log_2 n \rfloor \to \infty$, which forces:

Consequently, $X_n \xrightarrow{P} 0$.

2. Demonstrate Failure of Almost Sure Convergence:

Pick any arbitrary point $\omega \in [0, 1)$. In each generation $k$, the collection of intervals $\{ [j/2^k, (j+1)/2^k) \}_{j=0}^{2^k - 1}$ forms a complete partition of $[0, 1)$. Therefore, for every single generation $k$, there is exactly one interval in that generation containing $\omega$, meaning $X_n(\omega) = 1$ for that index $n$. Conversely, the remaining $2^k - 1$ intervals do not contain $\omega$, so $X_n(\omega) = 0$ for those indices. As $k \to \infty$, the point $\omega$ is struck infinitely many times by intervals containing it, and avoided infinitely many times. Therefore, the numerical sequence of real numbers $\{X_n(\omega)\}_{n=1}^\infty$ alternates endlessly between 0 and 1:

Because the sequence oscillates perpetually, the limit $\lim_{n \to \infty} X_n(\omega)$ does not exist for any $\omega \in [0, 1)$! The set of convergence points is the empty set:

Thus, $X_n$ fails to converge almost surely to 0. In fact, it diverges almost everywhere!

Counterexample 2: Almost Sure Convergence and Convergence in Probability Do NOT Imply $L^r$ Convergence (The "Escape to Infinite Energy" Sequence)

Let $\Omega = (0, 1)$ with uniform Lebesgue measure. Define the sequence of random variables:

where $r \ge 1$ is an integer. Let $X \equiv 0$.

1. Verify Almost Sure Convergence:

For any fixed outcome $\omega \in (0, 1)$, by Archimedes' principle there exists an integer $N_\omega = \lceil 1/\omega \rceil$. For all indices $n > N_\omega$, we have $1/n < \omega$, which implies $\omega \notin (0, 1/n)$. Thus, $X_n(\omega) = 0$ for all $n > N_\omega$. Consequently, for every $\omega \in (0, 1)$:

Because pointwise convergence holds on the entire sample space $\Omega$ (which has probability 1), we have $X_n \xrightarrow{\text{a.s.}} 0$. By Theorem 7.1, $X_n \xrightarrow{P} 0$ as well.

2. Evaluate the $L^r$ Norm:

Compute the expected $r$-th power deviation:

Taking the limit as $n \to \infty$:

Far from converging to 0, the expected $r$-th power diverges to infinity! Even though the probability mass of the spike collapses to zero ($P(X_n \ne 0) = 1/n \to 0$), the height of the spike grows so aggressively ($n^{2/r}$) that the integrated energy blows up.

Counterexample 3: Convergence in Distribution Does NOT Imply Convergence in Probability

Let $X \sim \mathcal{N}(0, 1)$ be a standard Gaussian random variable defined on $(\Omega, \mathcal{F}, P)$. For all $n \ge 1$, define:

1. Verify Convergence in Distribution:

By the symmetry of the Gaussian density around zero:

Thus, for every $n$, $X_n$ has the exact same cumulative distribution function as $X$. Trivially, $X_n \xrightarrow{d} X$.

2. Test Convergence in Probability:

Consider the difference $X_n - X = -X - X = -2X$. Since $X \sim \mathcal{N}(0, 1)$, $-2X \sim \mathcal{N}(0, 4)$. Choose $\epsilon = 1$. The probability of deviation is constant across all $n$:

Taking the limit as $n \to \infty$:

The sequence completely fails to converge in probability to $X$. Convergence in distribution merely asserts that the probability distributions match in the limit; it carries zero information about the joint co-dependence or physical proximity of the random variables on individual sample trajectories.

Important Partial Converses

Under specific structural conditions, the reverse implications do hold:

1. Convergence in Distribution to a Degenerate Constant:

If $X_n \xrightarrow{d} c$, where $c \in \mathbb{R}$ is a non-random constant, then $X_n \xrightarrow{P} c$. Proof: For any $\epsilon > 0$, $P(|X_n - c| \ge \epsilon) = P(X_n \le c - \epsilon) + P(X_n \ge c + \epsilon)$. Since $c$ is a constant, the limiting CDF $F(x) = \mathbf{1}_{[c, \infty)}(x)$ is continuous at both $c - \epsilon$ and $c + \epsilon$. As $n \to \infty$, $P(X_n \le c - \epsilon) \to F(c - \epsilon) = 0$ and $P(X_n \le c + \epsilon/2) \to F(c + \epsilon/2) = 1$. Consequently, $P(|X_n - c| \ge \epsilon) \to 0$.

2. Riesz's Subsequential Theorem:

If $X_n \xrightarrow{P} X$, there exists a strictly increasing subsequence of indices $\{n_k\}_{k=1}^\infty$ such that $X_{n_k} \xrightarrow{\text{a.s.}} X$.

3. Vitali's Convergence Theorem:

If $X_n \xrightarrow{P} X$, then $X_n \xrightarrow{L^r} X$ if and only if the family of random variables $\{|X_n|^r\}_{n=1}^\infty$ is uniformly integrable, meaning:

The proof of limit theorems in probability theory relies on bounding the probability that a random variable deviates into extreme tail regions using information about its moments.

1. Markov's Inequality

Markov's inequality provides the foundational bound connecting probabilities to first moments of non-negative random variables.

Theorem (Markov's Inequality): Let $Y$ be a non-negative random variable ($Y \ge 0$ almost surely) with finite expectation $\mathbb{E}[Y] < \infty$. Then for any arbitrary positive constant $a > 0$:

Proof: We define the indicator random variable $\mathbf{1}_{\{Y \ge a\}}$, which equals 1 when $Y \ge a$ and 0 when $Y < a$. Because $Y \ge 0$ everywhere, we have the elementary pointwise inequality:

To verify this, consider two cases:

• If $Y(\omega) < a$, then $a \cdot \mathbf{1}_{\{Y \ge a\}}(\omega) = 0 \le Y(\omega)$ since $Y \ge 0$.
• If $Y(\omega) \ge a$, then $a \cdot \mathbf{1}_{\{Y \ge a\}}(\omega) = a \le Y(\omega)$.

Taking expectations on both sides, and using the monotonicity and linearity of expectation:

Dividing both sides by the positive scalar $a > 0$ yields Markov's inequality:

2. Chebyshev's Inequality

Chebyshev's inequality extends Markov's bound to general random variables with finite variance by applying Markov's inequality to squared centered deviations.

Theorem (Chebyshev's Inequality): Let $X$ be any random variable with finite mean $\mu = \mathbb{E}[X]$ and finite variance $\sigma^2 = \text{Var}(X) < \infty$. Then for any positive constant $\epsilon > 0$:

Proof: Notice that the event $\{|X - \mu| \ge \epsilon\}$ is identical to the event $\{(X - \mu)^2 \ge \epsilon^2\}$. The variable $Y = (X - \mu)^2$ is non-negative almost surely, and its expectation is $\mathbb{E}[Y] = \mathbb{E}[(X - \mu)^2] = \sigma^2$. Applying Markov's inequality to $Y$ with threshold $a = \epsilon^2$:

Writing $\epsilon = k\sigma$ where $k > 0$ represents the number of standard deviations from the mean:

This universal bound holds for any probability distribution regardless of shape, skewness, or kurtosis. It guarantees that at least $75\%$ of the probability mass must lie within $2\sigma$ of the mean ($k=2$), at least $88.89\%$ within $3\sigma$ ($k=3$), and at least $96\%$ within $5\sigma$ ($k=5$).

3. Cantelli's One-Sided Chebyshev Inequality

Chebyshev's inequality bounds deviations symmetrically on both sides of the mean. If one is interested in a one-sided tail probability $P(X - \mu \ge \epsilon)$, Cantelli's inequality delivers a substantially sharper bound.

Theorem (Cantelli's Inequality): Let $X$ have mean $\mu$ and variance $\sigma^2 < \infty$. For any $\epsilon > 0$:

For instance, for $\epsilon = 2\sigma$, two-sided Chebyshev yields $P(|X - \mu| \ge 2\sigma) \le 1/4 = 0.25$, whereas Cantelli gives $P(X - \mu \ge 2\sigma) \le \sigma^2 / (\sigma^2 + 4\sigma^2) = 1/5 = 0.20$.

4. Jensen's Inequality

Jensen's inequality relates the expectation of a transformed random variable to the transform of its expectation when the transforming function is convex.

Theorem (Jensen's Inequality): Let $g: \mathbb{R} \to \mathbb{R}$ be a convex function, and let $X$ be a random variable such that $\mathbb{E}[|X|] < \infty$ and $\mathbb{E}[|g(X)|] < \infty$. Then:

Furthermore, if $g$ is strictly convex on an interval containing the support of $X$, equality holds if and only if $X$ is constant almost surely ($P(X = \mathbb{E}[X]) = 1$).

Proof using Supporting Hyperplanes: Recall that a convex function $g(x)$ lies above all of its tangent supporting lines. That is, at any point $\mu \in \mathbb{R}$, there exists a real slope $m \in \mathbb{R}$ (belonging to the subgradient of $g$ at $\mu$) such that:

Set $\mu = \mathbb{E}[X]$ and substitute the random variable $X$ for $x$:

Taking expectations of both sides:

This establishes the inequality. $\blacksquare$

5. Chernoff Bounding Technique and Exponential Inequalities

While Chebyshev's inequality utilizes second moments to yield polynomial decay bounds ($O(1/\epsilon^2)$), the Chernoff bound uses the entire moment generating function to yield exponential decay bounds ($O(e^{-c \epsilon^2})$).

Let $X$ possess a moment generating function $M_X(t) = \mathbb{E}[e^{tX}]$ defined in a neighborhood containing $t > 0$. For any threshold $a > 0$ and any parameter $t > 0$, the event $\{X \ge a\}$ is identical to $\{e^{tX} \ge e^{ta}\}$. Applying Markov's inequality:

Because this bound is valid for every $t > 0$, we can optimize over $t$ to find the tightest possible upper bound:

The function $\Lambda^*(a) = \sup_{t > 0} \{ta - \log M_X(t)\}$ is known as the Fenchel-Legendre transform (or convex conjugate) of the cumulant generating function. This exponential decay rate forms the bedrock of Cramér's Large Deviation Theory and modern non-asymptotic concentration of measure.

The Law of Large Numbers is the philosophical and mathematical cornerstone of statistical theory: it asserts that the empirical average computed from repeated independent observations converges to the underlying population expectation. The "Weak" Law specifically formalizes convergence in probability:

WLLN under Finite Variance: Chebyshev's Theorem

We begin with the classical theorem formulated by Chebyshev in 1867, which assumes finite second moments but requires only pairwise uncorrelated random variables.

Theorem 7.5 (Chebyshev's WLLN): Let $\{X_i\}_{i=1}^\infty$ be a sequence of pairwise uncorrelated random variables with common expectation $\mathbb{E}[X_i] = \mu$ and uniformly bounded variances $\text{Var}(X_i) = \sigma_i^2 \le \sigma^2 < \infty$. Let $\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i$. Then for every $\epsilon > 0$:

That is, $\bar{X}_n \xrightarrow{P} \mu$.

Proof: Compute the expectation of the sample mean using linearity:

Next, compute the variance of the sample mean. Because the variables are pairwise uncorrelated, $\text{Cov}(X_i, X_j) = 0$ for all $i \ne j$. The variance of the sum equals the sum of the variances:

Now apply Chebyshev's inequality to the random variable $\bar{X}_n$:

Fix any arbitrary error threshold $\epsilon > 0$. Taking the limit as $n \to \infty$:

Because probabilities are bounded below by 0, the squeeze theorem forces the limit to be zero. Hence $\bar{X}_n \xrightarrow{P} \mu$. $\blacksquare$

Khinchin's Weak Law of Large Numbers (Eliminating the Variance Assumption)

Chebyshev's proof relied critically upon the existence of the variance $\sigma^2 < \infty$. In 1929, the Soviet mathematician Aleksandr Khinchin achieved a major breakthrough by proving that the finite variance condition is completely unnecessary for i.i.d. observations: the existence of the first moment $\mathbb{E}[|X_i|] < \infty$ is fully sufficient.

Theorem 7.6 (Khinchin's WLLN): Let $\{X_i\}_{i=1}^\infty$ be a sequence of independent and identically distributed (i.i.d.) random variables with finite expectation $\mathbb{E}[X_i] = \mu$. Then:

Proof using Characteristic Functions: Let $\phi_X(t) = \mathbb{E}[e^{itX}]$ denote the characteristic function of $X_1$. Since $\mathbb{E}[|X_1|] < \infty$, the characteristic function is continuously differentiable on $\mathbb{R}$ with first derivative at the origin:

Applying Taylor's theorem with Peano's remainder to $\phi_X(t)$ in a neighborhood of $t = 0$:

Now, consider the characteristic function of the sample mean $\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i$:

By the independence of $X_1, \dots, X_n$, the expectation of the product factors into the product of expectations:

For any fixed $t \in \mathbb{R}$, as $n \to \infty$, the argument $t/n \to 0$. Substituting the Taylor expansion of $\phi_X(t/n)$:

Recall the classical real analysis limit identity: if $a_n \to a \in \mathbb{C}$, then $\lim_{n \to \infty} (1 + a_n/n)^n = e^a$. Here, letting $a_n = i \mu t + n \cdot o(t/n) \to i \mu t$ as $n \to \infty$:

Observe that $\phi_c(t) = e^{ict}$ is the characteristic function of the degenerate constant random variable taking value $c = \mu$ with probability 1.

By Lévy's Continuity Theorem (Section 7.6), because the sequence of characteristic functions $\phi_{\bar{X}_n}(t)$ converges pointwise to the characteristic function of the constant $\mu$, the sequence of random variables converges in distribution to $\mu$:

Finally, by the partial converse established in Section 7.2 (convergence in distribution to a constant implies convergence in probability), we conclude:

Failure of the WLLN: The Cauchy Distribution

It is instructive to understand what happens when the first moment does not exist. Consider an i.i.d. sequence $X_1, X_2, \dots \sim \text{Cauchy}(0, 1)$ with probability density function:

The integral $\int_{-\infty}^\infty |x| f(x) \, dx = \frac{2}{\pi}\int_0^\infty \frac{x}{1+x^2} \, dx = \lim_{M \to \infty} \frac{1}{\pi} \ln(1 + M^2) = +\infty$. Thus, the Cauchy distribution has no mean.

The characteristic function of a standard Cauchy variable is $\phi_X(t) = e^{-|t|}$. The characteristic function of the sample mean $\bar{X}_n$ is:

Remarkably, the characteristic function of $\bar{X}_n$ is identical to the characteristic function of a single observation $X_1$ for all sample sizes $n \ge 1$! By the uniqueness of characteristic functions, the sample mean $\bar{X}_n$ of $1,000,000$ Cauchy observations has the exact same standard Cauchy distribution as a single observation. The sample average never settles down or concentrates; it fluctuates wildly over the real line forever, violating the Weak Law of Large Numbers.

While the Weak Law guarantees that the probability of significant error at index $n$ vanishes, it leaves open the possibility that an observer watching an infinite sequence of data might see erratic excursions recur indefinitely into the future. The Strong Law of Large Numbers rules this out completely: it establishes that with probability 1, the sample average path converges pointwise to the true expectation $\mu$.

Kolmogorov's Strong Law of Large Numbers

Theorem 7.7 (Kolmogorov's SLLN): Let $\{X_i\}_{i=1}^\infty$ be a sequence of independent and identically distributed (i.i.d.) random variables.

1. If $\mathbb{E}[|X_1|] < \infty$, then:

where $\mu = \mathbb{E}[X_1]$.

1. Conversely, if $P(\lim_{n \to \infty} \bar{X}_n = c) = 1$ for some finite constant $c \in \mathbb{R}$, then necessarily $\mathbb{E}[|X_1|] < \infty$ and $c = \mathbb{E}[X_1]$. If $\mathbb{E}[|X_1|] = \infty$, then $\limsup_{n \to \infty} |\bar{X}_n| = \infty$ almost surely.

Foundational Machinery: The Borel-Cantelli Lemmas

The primary analytical engine for proving almost sure convergence is the pair of lemmas established by Émile Borel (1909) and Francesco Cantelli (1917).

Let $\{E_n\}_{n=1}^\infty$ be an infinite sequence of events in $(\Omega, \mathcal{F}, P)$. The event that infinitely many $E_n$ occur (abbreviated $E_n \text{ i.o.}$) is defined as:

Lemma 7.1 (First Borel-Cantelli Lemma): Let $\{E_n\}_{n=1}^\infty$ be any sequence of events (no independence required). If the sum of their probabilities converges:

then the probability that infinitely many events occur is zero:

Proof: For each $m \ge 1$, let $B_m = \bigcup_{n=m}^\infty E_n$. Since $\{B_m\}$ is decreasing, $\{E_n \text{ i.o.}\} = \bigcap_{m=1}^\infty B_m$. By countable subadditivity of probability measures:

Because the infinite series $\sum_{n=1}^\infty P(E_n)$ converges, its tail sum must vanish as $m \to \infty$:

By the continuity of probability from above:

Since probabilities are non-negative, $P(E_n \text{ i.o.}) = 0$. $\blacksquare$

Lemma 7.2 (Second Borel-Cantelli Lemma - The Independent Converse): Let $\{E_n\}_{n=1}^\infty$ be a sequence of mutually independent events. If the sum of their probabilities diverges:

then the probability that infinitely many events occur is one:

Kronecker's Lemma and Kolmogorov's Convergence Criterion

To establish the Strong Law for sums of independent random variables, Kolmogorov utilized a remarkable real-analysis summation-by-parts identity.

Lemma 7.3 (Kronecker's Lemma): Let $\{x_n\}_{n=1}^\infty$ be a sequence of real numbers such that the infinite series $\sum_{n=1}^\infty x_n$ converges to a finite limit. Let $\{b_n\}_{n=1}^\infty$ be a positive, strictly increasing sequence of real numbers with $b_n \uparrow \infty$. Then:

Theorem 7.8 (Kolmogorov's Criterion for Independent Series): Let $\{Y_n\}_{n=1}^\infty$ be independent random variables with zero means $\mathbb{E}[Y_n] = 0$ and variances $\text{Var}(Y_n) = \sigma_n^2$. If:

then $\frac{1}{n}\sum_{k=1}^n Y_k \xrightarrow{\text{a.s.}} 0$.

The Truncation Technique

In proving the SLLN for i.i.d. random variables with only $\mathbb{E}[|X_1|] < \infty$, the challenge is that the original variables $X_n$ may have infinite variance ($\sigma^2 = \infty$), preventing direct application of Kolmogorov's criterion. Kolmogorov overcame this obstacle with the ingenious truncation method.

Define the truncated variables:

1. Equivalence of Tails: Show that $P(X_n \ne Y_n \text{ i.o.}) = 0$.

Observe that $P(X_n \ne Y_n) = P(|X_n| > n) = P(|X_1| > n)$. Using the integral identity for non-negative random variables:

By the First Borel-Cantelli Lemma, the events $\{X_n \ne Y_n\}$ occur only finitely many times with probability 1! Consequently, the sequences $\frac{1}{n}\sum_{i=1}^n X_i$ and $\frac{1}{n}\sum_{i=1}^n Y_i$ have the exact same almost sure limiting behavior.

2. Sum of Variances of Truncated Variables:

Compute the variance sum:

Exchanging summation and expectation by Fubini's theorem:

Since $\sum_{n=m}^\infty \frac{1}{n^2} \le \frac{2}{m}$ for all $m \ge 1$:

Because the series of variances converges, Kolmogorov's criterion and Kronecker's lemma imply that:

3. Convergence of Truncated Expectations:

By the Dominated Convergence Theorem, $\lim_{n \to \infty} \mathbb{E}[Y_n] = \lim_{n \to \infty} \mathbb{E}[X_1 \mathbf{1}_{\{|X_1| \le n\}}] = \mathbb{E}[X_1] = \mu$. By Cesàro summation: $\lim_{n \to \infty} \frac{1}{n}\sum_{i=1}^n \mathbb{E}[Y_i] = \mu$. Combining these results yields the complete Strong Law:

The ultimate analytical weapon for establishing convergence in distribution is the characteristic function.

Definition and Fundamental Properties

Definition: For any real-valued random variable $X$ with cumulative distribution function $F(x)$, its characteristic function $\phi_X: \mathbb{R} \to \mathbb{C}$ is the Fourier-Stieltjes transform of its probability distribution:

where $i = \sqrt{-1}$ is the imaginary unit and $t \in \mathbb{R}$ is the frequency parameter.

Unlike the Moment Generating Function $M_X(t) = \mathbb{E}[e^{tX}]$, which requires exponential tails and diverges to infinity for distributions with heavy tails (such as the Student's $t$, Cauchy, or Pareto distributions), the characteristic function always exists for every random variable:

Key structural properties include:

1. Normalization: $\phi_X(0) = \mathbb{E}[e^0] = 1$.

2. Hermitian Symmetry: $\phi_X(-t) = \overline{\phi_X(t)}$, where $\overline{z}$ denotes the complex conjugate.

3. Uniform Continuity: $\phi_X(t)$ is uniformly continuous on the entire real line $\mathbb{R}$.

4. Affine Transformation: If $Y = aX + b$ for constants $a, b \in \mathbb{R}$, then:

5. Convolution of Independent Variables: If $X_1, X_2, \dots, X_n$ are independent, then:

The characteristic function converts tedious probability convolutions into simple algebraic multiplication.

Lévy's Inversion Theorem and Uniqueness

A characteristic function uniquely determines the underlying probability distribution.

Theorem 7.9 (Lévy's Inversion Formula): Let $F(x)$ denote the CDF of $X$, and let $\phi(t)$ denote its characteristic function. For any two continuity points $a < b$ of $F$:

Furthermore, if the characteristic function is absolutely integrable ($\int_{-\infty}^\infty |\phi(t)| \, dt < \infty$), the distribution possesses a continuous probability density function $f(x)$ given by the inverse Fourier transform:

Lévy's Continuity Theorem

Lévy's Continuity Theorem provides the bridge connecting the pointwise convergence of characteristic functions to convergence in distribution of random variables.

Theorem 7.10 (Lévy's Continuity Theorem): Let $\{X_n\}_{n=1}^\infty$ be a sequence of random variables with characteristic functions $\phi_n(t) = \mathbb{E}[e^{itX_n}]$, and let $X$ have characteristic function $\phi(t)$.

1. If $X_n \xrightarrow{d} X$, then $\phi_n(t) \to \phi(t)$ pointwise for every $t \in \mathbb{R}$ as $n \to \infty$. (In fact, the convergence is uniform on every compact interval $[-K, K]$).
2. Conversely, suppose $\phi_n(t)$ converges pointwise to a limiting function:

If the limit function $g(t)$ is continuous at the origin $t = 0$, then $g(t)$ is the characteristic function of some random variable $X$ ($g(t) = \phi_X(t)$), and:

The continuity of $g(t)$ at $t = 0$ is essential: it prevents probability mass from escaping to $\pm \infty$ (a phenomenon known in measure theory as tightness).

The Central Limit Theorem (CLT) is indisputably one of the most remarkable discoveries in all of mathematics. It explains why the bell-shaped Gaussian distribution appears ubiquitously across astronomy, thermodynamics, genetics, biometrics, economics, and psychology: whenever an observed phenomenon is the cumulative sum of numerous small, independent random disturbances, the aggregate distribution converges inexorably toward a Gaussian law, regardless of the shape of the individual component distributions.

Formal Statement

Theorem 7.11 (Lindeberg-Lévy CLT): Let $\{X_i\}_{i=1}^\infty$ be an i.i.d. sequence of random variables with common expectation $\mu = \mathbb{E}[X_i]$ and finite variance $\sigma^2 = \text{Var}(X_i) \in (0, \infty)$. Define the standardized sample sum:

Then, as the sample size $n \to \infty$, $Z_n$ converges in distribution to a standard Gaussian random variable $\mathcal{N}(0, 1)$:

That is, for every real number $z \in \mathbb{R}$:

Step-by-Step Rigorous Proof

Proof: Without loss of generality, we can center the variables. Let $Y_i = \frac{X_i - \mu}{\sigma}$. Then $\{Y_i\}_{i=1}^\infty$ are i.i.d. random variables satisfying:

The standardized sample sum $Z_n$ can be written cleanly as:

Let $\phi_Y(t) = \mathbb{E}[e^{itY_1}]$ denote the characteristic function of $Y_1$. Since $Y_1$ possesses finite first and second moments ($\mathbb{E}[|Y_1|^2] = 1 < \infty$), Taylor's theorem with Peano's form of the remainder implies that around $t = 0$:

Evaluating the derivatives at the origin:

• $\phi_Y(0) = 1$
• $\phi_Y'(0) = i \, \mathbb{E}[Y_1] = 0$
• $\phi_Y''(0) = i^2 \, \mathbb{E}[Y_1^2] = -1 \cdot 1 = -1$

Substituting these values into the Taylor expansion yields:

Now, consider the characteristic function of $Z_n = \sum_{i=1}^n (Y_i / \sqrt{n})$:

For any fixed $t \in \mathbb{R}$, as $n \to \infty$, the scaled argument $t/\sqrt{n} \to 0$. Substituting the expansion:

We examine the natural logarithm of $\phi_{Z_n}(t)$. Using the Taylor series expansion for the complex logarithm $\ln(1 + z) = z + O(|z|^2)$ for $|z| < 1/2$:

Taking the limit as $n \to \infty$:

Exponentiating both sides:

Notice that $\phi(t) = e^{-t^2/2}$ is precisely the characteristic function of the standard Gaussian distribution $\mathcal{N}(0, 1)$.

Since the limiting function $e^{-t^2/2}$ is continuous at $t = 0$ (with $e^0 = 1$), Lévy's Continuity Theorem (Theorem 7.10) asserts that:

In many realistic econometric, engineering, and experimental environments, observations are neither identically distributed nor collected in an unchanging sequence. For example, in regression analysis, the explanatory design points $x_i$ vary across observations, so each error contribution has a different scale. To handle non-identically distributed and dependent structures, probability theorists developed the framework of triangular arrays.

Definition of a Triangular Array

A triangular array of random variables is a doubly indexed collection:

where for each row $n \ge 1$, the random variables $\{X_{ni}\}_{i=1}^{k_n}$ (with $k_n \to \infty$) are mutually independent with means $\mathbb{E}[X_{ni}] = 0$ and variances $\sigma_{ni}^2 = \text{Var}(X_{ni})$. Let the total row variance be normalized to unity:

We seek conditions under which the row sum $S_n = \sum_{i=1}^{k_n} X_{ni} \xrightarrow{d} \mathcal{N}(0, 1)$.

The Lindeberg Condition

In 1922, Finnish mathematician Jarl Waldemar Lindeberg introduced a condition that restricts the contribution of extreme tail outliers.

Definition (Lindeberg Condition): The triangular array satisfies the Lindeberg Condition if for every fixed threshold $\epsilon > 0$:

In intuitive terms, the Lindeberg condition requires that the expected squared energy residing in the tails beyond $\epsilon$ shrinks to zero across the entire sum: no individual term is permitted to dominate the variance of the aggregate sum.

Theorem 7.12 (Lindeberg-Feller Central Limit Theorem): Let $\{X_{ni}\}$ be an independent triangular array with $\mathbb{E}[X_{ni}] = 0$ and $\sum_{i=1}^{k_n} \sigma_{ni}^2 = 1$.

1. Lindeberg's Theorem: If the Lindeberg condition holds, then:

Furthermore, the array is Uniformly Asymptotically Negligible (UAN):

2. Feller's Converse (1935): If $S_n \xrightarrow{d} \mathcal{N}(0, 1)$ AND the UAN condition holds, then the Lindeberg condition is necessarily satisfied!

Together, Lindeberg and Feller proved that under the natural UAN condition, the Lindeberg condition is not merely sufficient, but necessary and sufficient for asymptotic normality.

Lyapunov's Condition: An Easily Verifiable Criterion

Checking the indicator expectation $\mathbb{E}[X_{ni}^2 \mathbf{1}_{\{|X_{ni}| \ge \epsilon\}}]$ can be analytically tedious. In 1901, Aleksandr Lyapunov provided a stronger, moments-based criterion that is far easier to verify in practice.

Theorem 7.13 (Lyapunov's CLT): Let $\{X_{ni}\}$ be an independent triangular array with $\mathbb{E}[X_{ni}] = 0$ and row variance $s_n^2 = \sum_{i=1}^{k_n} \sigma_{ni}^2$. If there exists some $\delta > 0$ such that:

then $S_n / s_n \xrightarrow{d} \mathcal{N}(0, 1)$.

Proof that Lyapunov Implies Lindeberg: Observe that whenever $|X_{ni}| \ge \epsilon s_n$, we have $|X_{ni}|^{\delta} \ge (\epsilon s_n)^\delta$, which implies:

Therefore:

Dividing both sides by $s_n^2$:

Thus, Lyapunov's condition automatically implies Lindeberg's condition! $\blacksquare$

The Central Limit Theorem guarantees that the CDF error $|F_n(z) - \Phi(z)|$ vanishes as $n \to \infty$. However, in scientific practice, sample size is always finite. How fast does convergence occur? How large must $n$ be before the Gaussian approximation can be safely trusted? The Berry-Esseen Theorem answers this question with precision.

The Berry-Esseen Bound

Theorem 7.14 (Berry-Esseen Theorem): Let $\{X_i\}_{i=1}^n$ be i.i.d. random variables with mean $\mu$, variance $\sigma^2 > 0$, and finite absolute third moment $\rho = \mathbb{E}[|X_1 - \mu|^3] < \infty$. Let $F_n(z) = P\left( \frac{\sqrt{n}(\bar{X}_n - \mu)}{\sigma} \le z \right)$. Then for all $z \in \mathbb{R}$ and all $n \ge 1$:

where $C$ is a universal, distribution-free mathematical constant.

Numerical Magnitude of the Constant $C$

The exact value of $C$ was sharpened through decades of harmonic analysis:

• Andrew Berry (1941) proved $C \le 1.88$.
• Carl-Gustav Esseen (1942) established $C \le 7.59$, and proved the fundamental lower bound $C \ge \frac{\sqrt{10} + 3}{6\sqrt{2\pi}} \approx 0.40973$.
• Irina Shevtsova (2011) proved the modern tight upper bound:

Thus, for any distribution with finite third moments:

Key Practical Insights from Berry-Esseen:

1. The $1/\sqrt{n}$ Convergence Rate: The uniform error between the true cumulative distribution and the normal CDF decays at rate $O(1/\sqrt{n})$. To cut the maximum error in half, the sample size must be quadrupled ($4n$).

2. The Role of Skewness: The ratio $\frac{\rho}{\sigma^3}$ measures the standardized third moment. For symmetric distributions (e.g., Uniform, Student's $t$ with $df > 3$), skewness $\gamma_1 = \mathbb{E}[(X-\mu)^3]/\sigma^3 = 0$, and convergence is remarkably swift (often $n \approx 15$ to $30$ yields excellent fidelity). For highly skewed distributions (e.g., Exponential where $\rho/\sigma^3 \approx 8.5$, or Lognormal where $\rho/\sigma^3 \gg 100$), much larger samples ($n \ge 500$) are required before normal approximations become reliable.

The limit theorems developed throughout this unit form the mathematical bedrock of classical and modern statistical inference.

1. Consistency of Sample Moments

Let $X_1, \dots, X_n$ be an i.i.d. sample from a population with finite $k$-th moment $\mu_k' = \mathbb{E}[X^k] < \infty$. The empirical $k$-th sample moment is defined as $m_k' = \frac{1}{n}\sum_{i=1}^n X_i^k$. By Kolmogorov's SLLN applied to the sequence of i.i.d. variables $Y_i = X_i^k$:

Furthermore, the sample variance $S_n^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar{X}_n)^2 = \frac{n}{n-1}\left( \frac{1}{n}\sum_{i=1}^n X_i^2 - (\bar{X}_n)^2 \right)$. By the SLLN, $\frac{1}{n}\sum X_i^2 \xrightarrow{\text{a.s.}} \mathbb{E}[X^2]$ and $\bar{X}_n \xrightarrow{\text{a.s.}} \mu$. By the Continuous Mapping Theorem:

The sample variance is a strongly consistent estimator of the population variance.

2. Large-Sample Confidence Intervals and Slutsky's Theorem

The Lindeberg-Lévy CLT guarantees that $\frac{\sqrt{n}(\bar{X}_n - \mu)}{\sigma} \xrightarrow{d} \mathcal{N}(0, 1)$. However, in practice, the true population standard deviation $\sigma$ is unknown. We substitute the sample standard deviation $S_n$.

By consistency, $S_n \xrightarrow{P} \sigma$, which implies $\frac{\sigma}{S_n} \xrightarrow{P} 1$. Applying Slutsky's Theorem:

This guarantees that the large-sample Wald confidence interval:

has exact asymptotic coverage probability $1 - \alpha$, regardless of the parent distribution of $X_i$!

3. Monte Carlo Integration and Simulation Error

Suppose we need to evaluate a complex multidimensional integral $I = \int_{[0, 1]^d} g(\mathbf{u}) \, d\mathbf{u}$. Monte Carlo integration samples $n$ independent uniform random vectors $\mathbf{U}_i \sim \text{Uniform}([0, 1]^d)$ and computes the empirical average:

By the Strong Law of Large Numbers, $\hat{I}_n \xrightarrow{\text{a.s.}} I$. By the Central Limit Theorem:

where $\sigma_g^2 = \text{Var}(g(\mathbf{U}))$. The absolute error $|\hat{I}_n - I| = O_P(n^{-1/2})$. Crucially, this convergence rate $n^{-1/2}$ is completely independent of the dimension $d$, allowing Monte Carlo algorithms to defeat the "curse of dimensionality" that cripples deterministic grid quadrature methods.