Normal-Sample Theory, Quadratic Forms, and Exact Sampling Distributions

The multivariate normal (Gaussian) distribution occupies a preeminent position in mathematical statistics. It serves not only as the exact sampling model for many physical measurements, but also as the universal limiting distribution for properly normalized sample sums and linear estimators via the multidimensional Central Limit Theorem.

Let $\mathbf{X} = (X_1, X_2, \dots, X_n)^T$ be an $n$-dimensional random vector defined on a probability space $(\Omega, \mathcal{F}, P)$. There are several mathematically equivalent ways to define the multivariate normal distribution:

1. Affine Transformation Definition (Constructive Definition): The random vector $\mathbf{X}$ is said to follow a multivariate normal distribution if there exists a $k$-dimensional vector of independent standard normal random variables $\mathbf{Z} = (Z_1, \dots, Z_k)^T \sim \mathcal{N}_k(\mathbf{0}, \mathbf{I}_k)$, an $n \times k$ deterministic real matrix $\mathbf{A}$, and an $n$-dimensional deterministic vector $\boldsymbol{\mu} \in \mathbb{R}^n$, such that:

Under this representation, the mean vector is $\mathbb{E}[\mathbf{X}] = \boldsymbol{\mu}$, and the covariance matrix is:

The resulting covariance matrix $\boldsymbol{\Sigma}$ is symmetric and positive semi-definite of rank at most $\min(n, k)$.

2. Linear Combination Definition (Cramér-Wold Device): The random vector $\mathbf{X}$ is multivariate normal if and only if for every deterministic constant vector $\mathbf{c} = (c_1, \dots, c_n)^T \in \mathbb{R}^n$, the scalar linear combination:

follows a univariate normal distribution:

(where a normal distribution with variance zero is understood as a degenerate point mass at its mean). This definition provides an immediate criterion for testing multivariate normality.

3. Characteristic Function Definition: The random vector $\mathbf{X} \sim \mathcal{N}_n(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ has joint characteristic function defined for all $\mathbf{t} \in \mathbb{R}^n$ by:

This characteristic function exists universally, regardless of whether $\boldsymbol{\Sigma}$ is non-singular or singular (degenerate).

The Non-Singular Probability Density Function

When the covariance matrix $\boldsymbol{\Sigma}$ is strictly positive-definite (denoted $\boldsymbol{\Sigma} \succ 0$), its determinant $\det(\boldsymbol{\Sigma}) > 0$ is strictly positive, and its matrix inverse $\boldsymbol{\Sigma}^{-1}$ exists. In this case, the multivariate normal distribution is absolutely continuous with respect to the $n$-dimensional Lebesgue measure, possessing the joint probability density function:

for all $\mathbf{x} \in \mathbb{R}^n$. The contours of constant probability density correspond to ellipsoids defined by the quadratic form $(\mathbf{x} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu}) = c^2$, known as Mahalanobis ellipsoids. The principal axes of these ellipsoids align precisely with the eigenvectors of $\boldsymbol{\Sigma}$, and their semi-axis lengths are proportional to the square roots of the corresponding eigenvalues.

Affine Invariance Property

A cornerstone property of the Gaussian family is its closure under affine transformations.

Theorem (Affine Invariance): Let $\mathbf{X} \sim \mathcal{N}_n(\boldsymbol{\mu}, \boldsymbol{\Sigma})$, and let $\mathbf{B}$ be an $m \times n$ real matrix and $\mathbf{d} \in \mathbb{R}^m$ a constant vector. Then the transformed vector $\mathbf{Y} = \mathbf{B} \mathbf{X} + \mathbf{d}$ follows an $m$-dimensional multivariate normal distribution:

Proof: Using the characteristic function definition, for any $\mathbf{s} \in \mathbb{R}^m$:

Substituting the known characteristic function of $\mathbf{X}$:

By the uniqueness theorem for characteristic functions, $\mathbf{Y}$ is uniquely identified as an $m$-variate normal vector with mean vector $\mathbf{B} \boldsymbol{\mu} + \mathbf{d}$ and covariance matrix $\mathbf{B} \boldsymbol{\Sigma} \mathbf{B}^T$. $\blacksquare$

In general probability theory, two random variables being uncorrelated ($\text{Cov}(X, Y) = 0$) does not imply that they are stochastically independent. A standard counterexample is $X \sim \mathcal{N}(0, 1)$ and $Y = X^2$: since the third moment of a standard normal is zero, $\text{Cov}(X, Y) = \mathbb{E}[X^3] - \mathbb{E}[X]\mathbb{E}[X^2] = 0$, yet $Y$ is a deterministic function of $X$ and thus completely dependent.

However, when random variables are jointly multivariate normal, the situation simplifies completely.

Theorem (Independence via Zero Covariance): Let $\mathbf{X} \sim \mathcal{N}_n(\boldsymbol{\mu}, \boldsymbol{\Sigma})$. Partition $\mathbf{X}$ into two sub-vectors $\mathbf{X}_1 \in \mathbb{R}^{n_1}$ and $\mathbf{X}_2 \in \mathbb{R}^{n_2}$ (with $n_1 + n_2 = n$), with corresponding partitioned mean vector and covariance matrix:

Then $\mathbf{X}_1$ and $\mathbf{X}_2$ are statistically independent if and only if their cross-covariance matrix vanishes identically:

Proof: If $\mathbf{X}_1$ and $\mathbf{X}_2$ are independent, then by definition $\text{Cov}(\mathbf{X}_1, \mathbf{X}_2) = \mathbf{0}$, establishing necessity. To prove sufficiency, suppose $\boldsymbol{\Sigma}_{12} = \mathbf{0}$ (which also implies $\boldsymbol{\Sigma}_{21} = \boldsymbol{\Sigma}_{12}^T = \mathbf{0}$). Then the covariance matrix is block-diagonal:

Let $\mathbf{t} = (\mathbf{t}_1^T, \mathbf{t}_2^T)^T \in \mathbb{R}^n$, where $\mathbf{t}_1 \in \mathbb{R}^{n_1}$ and $\mathbf{t}_2 \in \mathbb{R}^{n_2}$. The joint characteristic function of $\mathbf{X}$ is:

Using the exponential addition law $\exp(a + b) = \exp(a) \exp(b)$, the characteristic function factorizes:

Because the joint characteristic function factorizes into the product of the marginal characteristic functions for all $\mathbf{t}_1$ and $\mathbf{t}_2$, the random vectors $\mathbf{X}_1$ and $\mathbf{X}_2$ are strictly independent. $\blacksquare$

This theorem has immense practical significance: to prove that two linear combinations or projections of a Gaussian random vector are statistically independent, it is necessary and sufficient to demonstrate that their covariance is zero.

In linear models, regression diagnostics, and analysis of variance, test statistics (such as sums of squared residuals, explained sums of squares, and Wald statistics) take the mathematical form of quadratic forms.

Let $\mathbf{X} \sim \mathcal{N}_n(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ be a Gaussian random vector, and let $\mathbf{A}$ be an $n \times n$ real symmetric matrix ($\mathbf{A} = \mathbf{A}^T$). A quadratic form in $\mathbf{X}$ is a scalar random variable defined by:

If $\mathbf{A}$ were not initially symmetric, we could replace it without changing the value of $Q$ by its symmetric part $\frac{1}{2}(\mathbf{A} + \mathbf{A}^T)$, since $\mathbf{X}^T \mathbf{A} \mathbf{X} = \mathbf{X}^T \left(\frac{\mathbf{A} + \mathbf{A}^T}{2}\right) \mathbf{X}$. Thus, symmetry of $\mathbf{A}$ is assumed throughout without loss of generality.

First and Second Moments of Quadratic Forms

To compute the expectation of $Q = \mathbf{X}^T \mathbf{A} \mathbf{X}$, we utilize the trace operator $\text{tr}(\cdot)$, which satisfies cyclic commutativity: $\text{tr}(\mathbf{B} \mathbf{C}) = \text{tr}(\mathbf{C} \mathbf{B})$ for any matrices of compatible dimensions. Because $Q$ is a scalar ($1 \times 1$ matrix), it equals its own trace:

Recalling that the second moment matrix is $\mathbb{E}[\mathbf{X} \mathbf{X}^T] = \boldsymbol{\Sigma} + \boldsymbol{\mu} \boldsymbol{\mu}^T$:

This moment formula holds universally for any random vector $\mathbf{X}$ with mean $\boldsymbol{\mu}$ and covariance $\boldsymbol{\Sigma}$, regardless of normality.

When $\mathbf{X}$ is Gaussian, higher-order moments can also be evaluated. The exact variance of the quadratic form is:

Furthermore, if $\mathbf{B}$ is another $n \times n$ symmetric matrix defining a second quadratic form $\mathbf{X}^T \mathbf{B} \mathbf{X}$, their covariance is:

The most foundational sampling distribution for quadratic forms is the Chi-Square distribution ($\chi^2$).

Derivation from the Standard Normal Distribution

Let $Z \sim \mathcal{N}(0, 1)$ be a standard univariate normal random variable. Define $Y = Z^2$. The cumulative distribution function of $Y$ for $y > 0$ is:

where $\Phi(z) = \int_{-\infty}^z \frac{1}{\sqrt{2\pi}} e^{-u^2/2} \, du$ is the standard normal CDF. Differentiating with respect to $y$ via the Leibniz rule:

Recalling that $\Gamma(1/2) = \sqrt{\pi}$, we can rewrite the normalizing constant:

This is precisely the probability density function of a Gamma distribution with shape parameter $\alpha = 1/2$ and rate parameter $\beta = 1/2$. This distribution is designated as the Chi-Square distribution with 1 degree of freedom, denoted $\chi_1^2$:

Sum of Independent Squared Standard Normal Variates

Now, let $Z_1, Z_2, \dots, Z_k$ be mutually independent standard normal random variables: $\mathbf{Z} \sim \mathcal{N}_k(\mathbf{0}, \mathbf{I}_k)$. Define the sum of their squares:

The moment generating function of $Z_i^2 \sim \chi_1^2$ is:

By mutual independence, the MGF of the sum $U = \sum_{i=1}^k Z_i^2$ is the product of the individual MGFs:

By the MGF Uniqueness Theorem, $U$ follows a Gamma distribution with shape $\alpha = k/2$ and rate $\beta = 1/2$. This distribution is the Chi-Square distribution with $k$ degrees of freedom, denoted $\chi_k^2$. Its probability density function is:

and zero for $u \le 0$.

Fundamental Properties of the $\chi_k^2$ Distribution

1. Mean and Variance:

2. Reproductive Property: If $U_1 \sim \chi_{k_1}^2$ and $U_2 \sim \chi_{k_2}^2$ are independent, then:

3. Asymptotic Normality: By the classical Central Limit Theorem, as the degrees of freedom $k \to \infty$:

We now establish the central theorem connecting quadratic forms in Gaussian vectors to the Chi-Square distribution.

Let $\mathbf{Z} \sim \mathcal{N}_n(\mathbf{0}, \mathbf{I}_n)$ be a vector of standard normal random variables, and let $\mathbf{A}$ be an $n \times n$ real symmetric matrix. When does the quadratic form $Q = \mathbf{Z}^T \mathbf{A} \mathbf{Z}$ follow an exact Chi-Square distribution?

Theorem (Chi-Square Distribution of Quadratic Forms): The quadratic form $Q = \mathbf{Z}^T \mathbf{A} \mathbf{Z}$ follows a central Chi-Square distribution with $r$ degrees of freedom ($Q \sim \chi_r^2$) if and only if the matrix $\mathbf{A}$ is idempotent of rank $r$:

Proof: Since $\mathbf{A}$ is a real symmetric matrix, the Spectral Theorem of linear algebra guarantees the existence of an orthogonal matrix $\mathbf{P}$ ($\mathbf{P}^T \mathbf{P} = \mathbf{I}_n$) such that:

where $\boldsymbol{\Lambda} = \text{diag}(\lambda_1, \lambda_2, \dots, \lambda_n)$ is the diagonal matrix of real eigenvalues of $\mathbf{A}$. Substitute this spectral decomposition into the quadratic form:

Define the transformed random vector $\mathbf{W} = \mathbf{P}^T \mathbf{Z}$. Because $\mathbf{P}$ is orthogonal ($\mathbf{P}^T \mathbf{P} = \mathbf{I}_n$), the vector $\mathbf{W}$ is an orthogonal transformation of standard normals. Its distribution is:

Thus, the components $W_1, W_2, \dots, W_n$ are themselves mutually independent standard normal random variables! Expanding the quadratic form in terms of $\mathbf{W}$:

Because $W_1^2, \dots, W_n^2$ are independent $\chi_1^2$ random variables, the moment generating function of $Q$ is:

For $Q$ to follow a $\chi_r^2$ distribution, its MGF must equal $(1 - 2t)^{-r/2}$. Equating the two expressions:

Squaring both sides and taking reciprocal squares:

This equality of polynomials in $t$ holds for all $t$ in a neighborhood of zero if and only if exactly $r$ of the eigenvalues $\lambda_i$ are equal to 1, and the remaining $n - r$ eigenvalues are equal to 0:

A symmetric matrix whose eigenvalues are exclusively 0 and 1 is, by definition, an orthogonal projection matrix (symmetric idempotent matrix), satisfying $\mathbf{A}^2 = \mathbf{A}$. The number of non-zero eigenvalues equals the rank: $\text{rank}(\mathbf{A}) = \text{tr}(\mathbf{A}) = r$. This completes the proof in both directions. $\blacksquare$

Independence of Quadratic Forms: Craig's Theorem

A companion question concerns when two quadratic forms $\mathbf{Z}^T \mathbf{A} \mathbf{Z}$ and $\mathbf{Z}^T \mathbf{B} \mathbf{Z}$ are statistically independent.

Theorem (Craig's Theorem): Let $\mathbf{Z} \sim \mathcal{N}_n(\mathbf{0}, \mathbf{I}_n)$, and let $\mathbf{A}$ and $\mathbf{B}$ be $n \times n$ real symmetric matrices. The quadratic forms $Q_1 = \mathbf{Z}^T \mathbf{A} \mathbf{Z}$ and $Q_2 = \mathbf{Z}^T \mathbf{B} \mathbf{Z}$ are statistically independent if and only if the matrix product vanishes:

(or equivalently, $\mathbf{B} \mathbf{A} = \mathbf{0}$, since $(\mathbf{A}\mathbf{B})^T = \mathbf{B}^T \mathbf{A}^T = \mathbf{B}\mathbf{A}$).

The condition $\mathbf{A} \mathbf{B} = \mathbf{0}$ means that the column space of $\mathbf{B}$ is contained within the null space of $\mathbf{A}$. The two projection matrices project the data vector onto mutually orthogonal subspaces of $\mathbb{R}^n$, ensuring complete stochastic independence between the resulting energy allocations.

In analysis of variance (ANOVA) and linear regression, the total sum of squares $\sum_{i=1}^n X_i^2 = \mathbf{X}^T \mathbf{X}$ is partitioned into several component sums of squares:

where each $\mathbf{A}_j$ is a symmetric matrix representing a specific experimental factor or residual sum of squares. Cochran's Theorem provides the definitive conditions under which these component quadratic forms are mutually independent Chi-Square variates.

Theorem (Cochran's Theorem): Let $\mathbf{Z} \sim \mathcal{N}_n(\mathbf{0}, \mathbf{I}_n)$, and suppose the identity matrix can be decomposed into a sum of $k$ real symmetric matrices:

where $\text{rank}(\mathbf{A}_j) = r_j$. Then the following conditions are mathematically equivalent:

1. $\sum_{j=1}^k r_j = n$ (the sum of ranks equals the ambient dimension $n$).
2. Each matrix $\mathbf{A}_j$ is idempotent: $\mathbf{A}_j^2 = \mathbf{A}_j$ for all $j \in \{1, \dots, k\}$.
3. The matrices are pairwise orthogonal: $\mathbf{A}_j \mathbf{A}_m = \mathbf{0}$ for all $j \ne m$.
4. The quadratic forms $Q_j = \mathbf{Z}^T \mathbf{A}_j \mathbf{Z}$ are mutually independent Chi-Square random variables:

Cochran's Theorem guarantees that when the degrees of freedom partition cleanly ($\sum r_j = n$), the resulting ANOVA sums of squares are automatically independent Chi-Square variables, providing the exact theoretical justification for $F$-tests in the analysis of variance.

We are now equipped to deliver the definitive proof of the celebrated property of Gaussian samples: the sample mean $\bar{X}$ and the sample variance $S^2$ are completely independent.

Let $X_1, X_2, \dots, X_n$ be an i.i.d. random sample from $\mathcal{N}(\mu, \sigma^2)$. In vector notation, $\mathbf{X} \sim \mathcal{N}_n(\mu \mathbf{1}_n, \sigma^2 \mathbf{I}_n)$.

Define the projection matrix onto the 1-dimensional subspace spanned by $\mathbf{1}_n$:

and define the centering projection matrix:

Observe that:

1. $\bar{X} = \frac{1}{n} \mathbf{1}_n^T \mathbf{X}$ is a linear transformation of $\mathbf{X}$.
2. The residual vector is $\mathbf{X} - \bar{X}\mathbf{1}_n = \mathbf{M} \mathbf{X}$.
3. The sample variance is $S^2 = \frac{1}{n-1} \mathbf{X}^T \mathbf{M} \mathbf{X} = \frac{1}{n-1} \|\mathbf{M}\mathbf{X}\|^2$.

Now, compute the covariance between the linear vector $\bar{X}\mathbf{1}_n = \mathbf{P}\mathbf{X}$ and the residual vector $\mathbf{M}\mathbf{X}$:

Evaluating the matrix product $\mathbf{P} \mathbf{M}$:

Since $\mathbf{X}$ is multivariate normal, and the linear transformations $\mathbf{P}\mathbf{X}$ and $\mathbf{M}\mathbf{X}$ have zero cross-covariance, they are stochastically independent random vectors!

Because $\bar{X}$ is a measurable function of $\mathbf{P}\mathbf{X}$ (specifically, $\bar{X} = \frac{1}{n}\mathbf{1}_n^T (\mathbf{P}\mathbf{X})$), and $S^2$ is a measurable function of $\mathbf{M}\mathbf{X}$ (specifically, $S^2 = \frac{1}{n-1} \|\mathbf{M}\mathbf{X}\|^2$), any functions of independent random vectors are themselves independent. Therefore:

Furthermore, since $\mathbf{M}$ is idempotent ($\mathbf{M}^2 = \mathbf{M}$) of rank $\text{tr}(\mathbf{M}) = n - 1$:

Thus, in a normal sample of size $n$:

1. $\bar{X} \sim \mathcal{N}\left(\mu, \frac{\sigma^2}{n}\right)$.
2. $\frac{(n-1)S^2}{\sigma^2} \sim \chi_{n-1}^2$.
3. $\bar{X}$ and $S^2$ are statistically independent.

In practical scientific inference, the true population variance $\sigma^2$ is rarely known. When an investigator replaces the unknown standard deviation $\sigma$ in the standardized mean $Z = \frac{\bar{X} - \mu}{\sigma / \sqrt{n}}$ with the sample standard deviation $S$, the resulting ratio is the Student's $t$-statistic:

General Definition of the $t$ Distribution

Let $Z \sim \mathcal{N}(0, 1)$ be a standard normal random variable, and let $V \sim \chi_\nu^2$ be a Chi-Square random variable with $\nu > 0$ degrees of freedom. Assume that $Z$ and $V$ are independent. The random variable:

is said to follow Student's $t$ distribution with $\nu$ degrees of freedom, denoted $T \sim t_\nu$.

Since in a Gaussian sample $Z = \frac{\bar{X} - \mu}{\sigma / \sqrt{n}} \sim \mathcal{N}(0, 1)$ and $V = \frac{(n-1)S^2}{\sigma^2} \sim \chi_{n-1}^2$ are independent, the ratio:

follows an exact $t_{n-1}$ distribution.

Analytical Derivation of the Probability Density Function

Because $Z$ and $V$ are independent, their joint density is the product of standard normal and Chi-Square densities:

Define the transformation:

The inverse transformation is:

The Jacobian matrix of the inverse mapping is:

The determinant of the Jacobian is:

Since $w > 0$, $|\det(J)| = w^{1/2} / \sqrt{\nu}$. The joint density of $(T, W)$ is:

To obtain the marginal density $f_T(t)$, integrate out $w$ over $(0, \infty)$:

Make the substitution $u = \frac{w}{2}\left(1 + \frac{t^2}{\nu}\right)$, so $w = \frac{2u}{1 + t^2/\nu}$ and $dw = \frac{2 \, du}{1 + t^2/\nu}$:

Substituting this back into $f_T(t)$ and canceling powers of 2:

This is the exact probability density function of Student's $t$ distribution with $\nu$ degrees of freedom.

Moments and Tail Decay

The $t$ distribution is symmetric about zero: $f_T(-t) = f_T(t)$. Therefore, all odd moments that exist are zero: $\mathbb{E}[T^k] = 0$ for odd $k < \nu$.

1. Mean: $\mathbb{E}[T] = 0$ if $\nu > 1$ (undefined for $\nu \le 1$).

2. Variance: $\text{Var}(T) = \frac{\nu}{\nu - 2}$ if $\nu > 2$ (infinite for $1 < \nu \le 2$).

3. Heavy Tails: As $|t| \to \infty$, $f_T(t) \sim C |t|^{-(\nu + 1)}$. The tails decay as a polynomial power law rather than exponentially, making the $t$ distribution much more prone to extreme values than the Gaussian distribution.

4. Gaussian Limit: Using Stirling's approximation, as $\nu \to \infty$:

recovering the standard normal density $\mathcal{N}(0, 1)$.

When comparing the variances of two independent Gaussian populations or evaluating the overall significance of a linear regression model, inference relies on the Snedecor-Fisher $F$ distribution.

General Definition of the $F$ Distribution

Let $U_1 \sim \chi_{\nu_1}^2$ and $U_2 \sim \chi_{\nu_2}^2$ be two independent Chi-Square random variables with $\nu_1$ and $\nu_2$ degrees of freedom, respectively. The random variable defined by the ratio of these variables scaled by their degrees of freedom:

is said to follow the $F$ distribution with $\nu_1$ and $\nu_2$ degrees of freedom, denoted $F \sim F_{\nu_1, \nu_2}$.

Analytical Derivation of the Density Function

Because $U_1$ and $U_2$ are independent, their joint density is:

Define the transformation:

The inverse transformation is:

The Jacobian determinant is:

The joint density of $(F, W)$ is:

Integrating out $w$ over $(0, \infty)$ using the Gamma integral yields the probability density function of $F$:

and zero for $f \le 0$.

Core Structural Properties of the $F$ Distribution

1. Reciprocal Property: If $F \sim F_{\nu_1, \nu_2}$, then its reciprocal satisfies:

This property allows critical values in the lower tail to be computed directly from upper-tail critical values:

2. Relation to Student's $t$: If $T \sim t_\nu$, then $T = \frac{Z}{\sqrt{V/\nu}}$, so:

The square of a Student's $t$ variate with $\nu$ degrees of freedom is an $F$ variate with $(1, \nu)$ degrees of freedom.

3. Moments:

Notice that the mean depends exclusively on the denominator degrees of freedom $\nu_2$.

When evaluating the power of statistical hypothesis tests under specific alternative hypotheses ($H_1$), test statistics no longer follow central sampling distributions. Instead, non-zero population effects induce non-central distributions.

The Non-Central Chi-Square Distribution

Let $X_1, X_2, \dots, X_k$ be independent normal random variables with unit variances but non-zero means: $X_i \sim \mathcal{N}(\mu_i, 1)$. Define the sum of their squares:

The random variable $Y$ follows a non-central Chi-Square distribution with $k$ degrees of freedom and non-centrality parameter $\lambda$, denoted $Y \sim \chi_k^2(\lambda)$, where:

The moment generating function of $Y$ is:

Expanding the exponential term via power series reveals that the non-central $\chi_k^2(\lambda)$ distribution is an exact Poisson mixture of central Chi-Square distributions:

where the mixing weights are Poisson probabilities with mean $\lambda / 2$, and the conditional components are central Chi-Square densities with degrees of freedom $k + 2j$. The mean and variance of $Y \sim \chi_k^2(\lambda)$ are:

The Non-Central Student's $t$ Distribution

Let $Z \sim \mathcal{N}(\delta, 1)$ have non-zero mean $\delta$, and let $V \sim \chi_\nu^2$ be independent of $Z$. The ratio:

follows a non-central Student's $t$ distribution with $\nu$ degrees of freedom and non-centrality parameter $\delta$, denoted $t_\nu(\delta)$. This distribution is essential for computing the statistical power of one-sample and two-sample Student's $t$-tests.

The Non-Central $F$ Distribution

Let $U_1 \sim \chi_{\nu_1}^2(\lambda)$ be a non-central Chi-Square variable, and let $U_2 \sim \chi_{\nu_2}^2$ be an independent central Chi-Square variable. The ratio:

follows a non-central $F$ distribution, denoted $F_{\nu_1, \nu_2}(\lambda)$. The non-central $F$ distribution governs the power calculations in ANOVA and general linear hypothesis testing.

Geometric Interpretation and Statistical Applications of Non-Central Distributions

In statistical testing, non-central distributions are the indispensable engine of power analysis and sample size determination. When the null hypothesis $H_0$ is false, the test statistic no longer follows a central distribution centered at zero or unity, but shifts toward the non-central distribution whose shape depends monotonically upon the non-centrality parameter $\lambda$.

For instance, in the classic analysis of variance (ANOVA) comparing $k$ experimental treatments with $n_i$ observations in the $i$-th group, the test statistic is the ratio of treatment mean square to error mean square:

Under the null hypothesis $H_0: \mu_1 = \mu_2 = \dots = \mu_k$, the numerator and denominator quadratic forms are central chi-squares, yielding the central $F(k-1, N-k)$ distribution. Under an alternative hypothesis $H_1$, the expected treatment means deviate from the grand mean, so the numerator becomes a non-central chi-square with non-centrality parameter:

where $\bar{\mu} = \frac{1}{N}\sum n_i \mu_i$. The test statistic follows an exact non-central $F$-distribution $F(k-1, N-k, \lambda)$. The statistical power of the test at significance level $\alpha$ is:

Because the cumulative distribution function of the non-central $F$ distribution is strictly decreasing in $\lambda$ for every fixed critical value, the power function $\text{Power}(\lambda)$ is strictly increasing in $\lambda$. This formalizes the mathematical intuition that power improves when:

1. The effect size $\sum n_i(\mu_i - \bar{\mu})^2$ is large;
2. The experimental error variance $\sigma^2$ is small;
3. The overall sample size $N$ is increased.

By inverting the non-central $F$ power function, researchers in clinical trials, agricultural science, and psychometrics can accurately calculate the required sample size to guarantee a target power (typically $80\%$ or $90\%$) prior to data collection.

In univariate normal sampling, the sum of squared deviations $\sum_{i=1}^n (X_i - \bar{X})^2$ follows a scaled Chi-Square distribution. In multivariate normal sampling, where each observation $\mathbf{X}_i$ is a $p$-dimensional random vector, the natural matrix analogue of the sum of squares is the scatter matrix $\sum_{i=1}^n (\mathbf{X}_i - \bar{\mathbf{X}})(\mathbf{X}_i - \bar{\mathbf{X}})^T$. The sampling distribution of this random matrix is the Wishart distribution, first derived by John Wishart in 1928.

Formal Definition of the Wishart Matrix

Let $\mathbf{Z}_1, \mathbf{Z}_2, \dots, \mathbf{Z}_m$ be an independent collection of $p$-dimensional multivariate normal vectors with zero mean vector and common $p \times p$ positive-definite covariance matrix $\boldsymbol{\Sigma}$:

The $p \times p$ random matrix formed by the sum of outer products:

is said to follow a central Wishart distribution with $m$ degrees of freedom and scale matrix $\boldsymbol{\Sigma}$, denoted:

Notice that when $p = 1$, the matrix collapses to a scalar $W = \sum_{i=1}^m Z_i^2$, where $Z_i \sim \mathcal{N}(0, \sigma^2)$, so $W / \sigma^2 \sim \chi_m^2$. Thus, the Wishart distribution represents a direct multidimensional tensor generalization of the Chi-Square law.

When the degrees of freedom satisfy $m \ge p$, the matrix $\mathbf{W}$ is non-singular and strictly positive-definite with probability 1. In that regular case, $\mathbf{W}$ possesses a matrix probability density function with respect to the Lebesgue measure on the cone of $p \times p$ symmetric positive-definite matrices:

where $\Gamma_p(a)$ is the multivariate Gamma function, defined as:

Properties and Bartlett's Decomposition

The Wishart distribution exhibits fundamental structural properties that parallel the univariate Chi-Square distribution:

1. Mean Matrix: The mathematical expectation of the random matrix is:

2. Reproductive Property: If $\mathbf{W}_1 \sim \mathcal{W}_p(m_1, \boldsymbol{\Sigma})$ and $\mathbf{W}_2 \sim \mathcal{W}_p(m_2, \boldsymbol{\Sigma})$ are independent, then:

3. Linear Invariance: If $\mathbf{C}$ is a non-singular $q \times p$ matrix, then:

4. Bartlett's Decomposition: If $\mathbf{W} \sim \mathcal{W}_p(m, \mathbf{I}_p)$ with $m \ge p$, then $\mathbf{W}$ can be factored uniquely as $\mathbf{W} = \mathbf{T} \mathbf{T}^T$, where $\mathbf{T}$ is a lower-triangular matrix with positive diagonal elements. Bartlett proved that all non-zero entries of $\mathbf{T}$ are mutually independent random variables, with squared diagonal elements following independent Chi-Square distributions ($T_{jj}^2 \sim \chi_{m - j + 1}^2$) and off-diagonal elements following independent standard normal distributions ($T_{jk} \sim \mathcal{N}(0, 1)$ for $j > k$). This decomposition provides an ultra-fast algorithm for simulating Wishart matrices in computational statistics.

In univariate analysis, Student's $t$-statistic tests hypotheses concerning a population mean when the variance is unknown. In $p$-dimensional multivariate settings, Harold Hotelling introduced the Hotelling's $T^2$ statistic to test hypotheses about the multivariate mean vector $\boldsymbol{\mu} \in \mathbb{R}^p$.

Let $\mathbf{X}_1, \dots, \mathbf{X}_n$ be an i.i.d. random sample of $p$-dimensional vectors from $\mathcal{N}_p(\boldsymbol{\mu}, \boldsymbol{\Sigma})$, with $n > p$. The sample mean vector and sample covariance matrix are:

Hotelling's $T^2$ statistic for testing the null hypothesis $H_0: \boldsymbol{\mu} = \boldsymbol{\mu}_0$ is defined as the squared Mahalanobis distance between the sample mean and the hypothesized mean, scaled by the sample size:

Exact Transformation to the Snedecor-Fisher $F$ Distribution

Under the null hypothesis $H_0: \boldsymbol{\mu} = \boldsymbol{\mu}_0$, the exact sampling distribution of Hotelling's $T^2$ can be transformed into an exact Snedecor-Fisher $F$ distribution.

Theorem (Hotelling's $T^2$ Representation): If $\mathbf{X}_1, \dots, \mathbf{X}_n \sim \mathcal{N}_p(\boldsymbol{\mu}_0, \boldsymbol{\Sigma})$ with $n > p$, then:

Proof Outline: From normal sampling theory, $\sqrt{n}(\bar{\mathbf{X}} - \boldsymbol{\mu}_0) \sim \mathcal{N}_p(\mathbf{0}, \boldsymbol{\Sigma})$, and $(n - 1)\mathbf{S} \sim \mathcal{W}_p(n - 1, \boldsymbol{\Sigma})$, with $\bar{\mathbf{X}}$ and $\mathbf{S}$ being completely independent. Standardizing by $\boldsymbol{\Sigma}^{-1/2}$, define $\mathbf{Z} = \sqrt{n}\boldsymbol{\Sigma}^{-1/2}(\bar{\mathbf{X}} - \boldsymbol{\mu}_0) \sim \mathcal{N}_p(\mathbf{0}, \mathbf{I}_p)$ and $\mathbf{W} = (n - 1)\boldsymbol{\Sigma}^{-1/2} \mathbf{S} \boldsymbol{\Sigma}^{-1/2} \sim \mathcal{W}_p(n - 1, \mathbf{I}_p)$. Then $T^2 = (n - 1) \mathbf{Z}^T \mathbf{W}^{-1} \mathbf{Z}$. Conditioning on $\mathbf{Z} = \mathbf{z}$, choose an orthogonal matrix $\mathbf{P}$ whose first row is $\mathbf{z}^T / \|\mathbf{z}\|$. By the invariance of the Wishart distribution under orthogonal rotations, the scalar quantity $\frac{\mathbf{z}^T \mathbf{W}^{-1} \mathbf{z}}{\|\mathbf{z}\|^2}$ is equal to the $(1, 1)$ entry of $(\mathbf{P} \mathbf{W} \mathbf{P}^T)^{-1}$. Using the block matrix inversion formula, the reciprocal of the $(1, 1)$ entry of an inverted standard Wishart matrix follows an independent Chi-Square distribution:

Therefore, conditional on $\mathbf{Z}$:

Multiplying by $\|\mathbf{Z}\|^2 \sim \chi_p^2$ (which is independent of $V$):

Dividing the numerator and denominator by their degrees of freedom:

This establishes the exact correspondence between Hotelling's $T^2$ and the $F$ distribution, enabling exact multivariate hypothesis testing across all dimensions. $\blacksquare$

In univariate statistics, the sample variance $S^2$ measures scalar dispersion. In multivariate analysis, dispersion must capture both the individual coordinate variances and their mutual correlations. A foundational scalar measure of multidimensional dispersion is the generalized sample variance, introduced by Samuel S. Wilks.

The generalized sample variance is defined as the determinant of the sample covariance matrix:

Geometric Interpretation as Hypervolume

The determinant $|\mathbf{S}|$ has an intuitive geometric meaning. Consider the sample vectors centered by the sample mean: $\mathbf{r}_i = \mathbf{X}_i - \bar{\mathbf{X}} \in \mathbb{R}^p$. These vectors span a $p$-dimensional parallelotope in Euclidean space. The volume of this parallelotope is directly proportional to $\sqrt{\det(\mathbf{S})}$.

• If the variables are completely uncorrelated and standardized, $\mathbf{S} = \mathbf{I}_p$, so $|\mathbf{S}| = 1$, representing maximal multidimensional spread.
• If one variable is a perfect linear combination of the other variables, the data points collapse onto a $(p - 1)$-dimensional hyperplane, the matrix $\mathbf{S}$ becomes singular, and $|\mathbf{S}| = 0$.

Exact Sampling Distribution of the Generalized Variance

Because $(n - 1)\mathbf{S} \sim \mathcal{W}_p(n - 1, \boldsymbol{\Sigma})$, the distribution of $|\mathbf{S}|$ is determined by the determinant of a Wishart matrix. Using Bartlett's decomposition, the determinant of $\mathbf{W} \sim \mathcal{W}_p(m, \mathbf{I}_p)$ decomposes into the product of the squared diagonal elements of the Cholesky factor $\mathbf{T}$:

Because the entries $T_{jj}$ are mutually independent with $T_{jj}^2 \sim \chi_{m - j + 1}^2$, the determinant of a Wishart matrix is distributed as the product of $p$ independent Chi-Square random variables:

Consequently, the generalized sample variance satisfies:

Taking natural logarithms transforms the product into a sum of independent log-Chi-Square random variables, which converges rapidly to a Gaussian distribution via the classical Central Limit Theorem as the sample size $n \to \infty$.

In many modern statistical applications—such as overparameterized regression models, spline smoothing, and spatial Markov random fields—the covariance matrix $\boldsymbol{\Sigma}$ of a normal vector $\mathbf{X} \sim \mathcal{N}_n(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ is singular ($\text{rank}(\boldsymbol{\Sigma}) = r < n$). In such degenerate settings, the classical inverse $\boldsymbol{\Sigma}^{-1}$ does not exist, and the probability distribution is concentrated entirely on an $r$-dimensional affine hyperplane within $\mathbb{R}^n$.

To construct quadratic forms and test statistics under singular covariance matrices, mathematical statistics deploys the Moore-Penrose pseudoinverse (or more generally, a generalized inverse $\boldsymbol{\Sigma}^-$).

A matrix $\boldsymbol{\Sigma}^-$ is a generalized inverse (or $g$-inverse) of $\boldsymbol{\Sigma}$ if it satisfies the fundamental Penrose condition:

The Moore-Penrose pseudoinverse, denoted $\boldsymbol{\Sigma}^+$, satisfies three additional conditions: $\boldsymbol{\Sigma}^+ \boldsymbol{\Sigma} \boldsymbol{\Sigma}^+ = \boldsymbol{\Sigma}^+$, $(\boldsymbol{\Sigma} \boldsymbol{\Sigma}^+)^T = \boldsymbol{\Sigma} \boldsymbol{\Sigma}^+$, and $(\boldsymbol{\Sigma}^+ \boldsymbol{\Sigma})^T = \boldsymbol{\Sigma}^+ \boldsymbol{\Sigma}$, making it unique.

Khatri's Theorem on Quadratic Forms in Singular Normal Vectors

The classical theorem on Chi-Square distributions of quadratic forms (Section 3.5) was generalized to singular normal vectors by C. G. Khatri in 1968.

Theorem (Khatri's Theorem): Let $\mathbf{X} \sim \mathcal{N}_n(\boldsymbol{\mu}, \boldsymbol{\Sigma})$, where $\boldsymbol{\Sigma}$ may be singular. Let $\mathbf{A}$ be an $n \times n$ real symmetric matrix. Then the quadratic form:

follows a non-central Chi-Square distribution $Q \sim \chi_r^2(\lambda)$ if and only if:

with degrees of freedom $r = \text{tr}(\mathbf{A} \boldsymbol{\Sigma})$ and non-centrality parameter $\lambda = \boldsymbol{\mu}^T \mathbf{A} \boldsymbol{\mu}$.

Furthermore, if $\boldsymbol{\mu} = \mathbf{0}$, the condition reduces to $\boldsymbol{\Sigma} \mathbf{A} \boldsymbol{\Sigma} \mathbf{A} \boldsymbol{\Sigma} = \boldsymbol{\Sigma} \mathbf{A} \boldsymbol{\Sigma}$, which simplifies in the non-singular case ($\boldsymbol{\Sigma} \succ 0$) to $\mathbf{A} \boldsymbol{\Sigma} \mathbf{A} = \mathbf{A}$, recovering the classical idempotency criterion. This theorem provides the complete mathematical foundation for testing linear hypotheses in rank-deficient design models.

Many elegant geometric and invariance properties of the multivariate normal distribution—such as the rotational symmetry of standard normal vectors and the quadratic structure of Mahalanobis contours—extend naturally to a broader, highly flexible class of probability distributions known as elliptically contoured (or elliptical) distributions.

Spherical Random Vectors

A continuous random vector $\mathbf{Z} \in \mathbb{R}^n$ is said to follow a spherical distribution if its probability distribution is invariant under all orthogonal transformations (rotations and reflections). That is, for every orthogonal matrix $\mathbf{P} \in \mathbb{R}^{n \times n}$ satisfying $\mathbf{P}^T \mathbf{P} = \mathbf{I}_n$:

When $\mathbf{Z}$ possesses a probability density function with respect to the Lebesgue measure, rotational invariance implies that the density $f_{\mathbf{Z}}(\mathbf{z})$ depends on the coordinate vector $\mathbf{z}$ solely through its squared Euclidean norm $\|\mathbf{z}\|^2 = \mathbf{z}^T \mathbf{z}$:

for some non-negative scalar function $g: [0, \infty) \to [0, \infty)$, termed the density generator.

A profound geometric decomposition characterizes all spherical distributions. Any spherical random vector $\mathbf{Z}$ can be uniquely represented in polar-coordinate form as:

where:

1. $R = \|\mathbf{Z}\| = \sqrt{\mathbf{Z}^T \mathbf{Z}} \ge 0$ is a non-negative scalar random variable representing the radial distance from the origin.
2. $\mathbf{U} = \mathbf{Z} / \|\mathbf{Z}\|$ is a random unit vector uniformly distributed over the $(n - 1)$-dimensional unit sphere $\mathbb{S}^{n-1} = \{\mathbf{u} \in \mathbb{R}^n : \|\mathbf{u}\| = 1\}$.
3. The radial distance $R$ and the spherical directional vector $\mathbf{U}$ are statistically independent!

This stochastic representation reveals that the direction $\mathbf{U}$ is purely geometric and completely universal across all spherical distributions, while the specific distribution within the spherical family is governed entirely by the choice of radial distribution for $R$.

Elliptical Random Vectors

The family of elliptical distributions is obtained by applying general affine transformations to spherical vectors. Let $\mathbf{Z} \in \mathbb{R}^k$ be a spherical random vector. Define:

where $\boldsymbol{\mu} \in \mathbb{R}^n$ is a location vector and $\mathbf{A} \in \mathbb{R}^{n \times k}$ is a dispersion matrix. Letting $\boldsymbol{\Sigma} = \mathbf{A} \mathbf{A}^T$, the random vector $\mathbf{X}$ is said to follow an elliptical distribution, denoted $\mathbf{X} \sim \mathcal{E}_n(\boldsymbol{\mu}, \boldsymbol{\Sigma}, g)$.

When $\boldsymbol{\Sigma}$ is positive-definite, the probability density function of $\mathbf{X}$ is given by:

Prominent examples of elliptical distributions include:

1. The Multivariate Normal Distribution: Generated by $g(u) = (2\pi)^{-n/2} e^{-u/2}$. In this special case, the radial variable squared $R^2 = \|\mathbf{Z}\|^2$ follows a Chi-Square distribution $\chi_n^2$. The normal distribution is the unique member of the spherical family whose components $Z_1, \dots, Z_n$ are mutually independent; for all other spherical distributions, rotational invariance forces coordinate dependence through the shared radial factor $R$.

2. The Multivariate Student's $t$ Distribution: Generated by $g(u) \propto (1 + u/\nu)^{-(\nu + n)/2}$. Here, the vector can be represented as $\mathbf{X} = \boldsymbol{\mu} + \frac{\mathbf{Z}}{\sqrt{V/\nu}}$, where $\mathbf{Z} \sim \mathcal{N}_n(\mathbf{0}, \boldsymbol{\Sigma})$ and $V \sim \chi_\nu^2$ is an independent scalar Chi-Square variable.

3. The Multivariate Cauchy Distribution: The special case of the multivariate $t$ distribution with $\nu = 1$.

4. The Multivariate Logistic and Laplace Distributions: Providing alternative symmetric heavy-tailed and double-exponential models.

Invariance of $F$ and $t$ Ratios under Spherical Distributions

A remarkable result in mathematical statistics, established by E. S. Andersen (1958) and K. T. Fang, is that many classical test statistics derived under normality remain exact invariants under any continuous spherical distribution!

Consider a statistic that takes the form of a scale-invariant ratio of homogeneous quadratic forms:

Substitute the stochastic polar representation $\mathbf{Z} = R \mathbf{U}$:

Notice that the radial random variable $R^2$ cancels completely from the numerator and denominator! Because $\mathbf{U}$ is uniformly distributed on the unit sphere for any spherical distribution, the sampling distribution of the ratio $T(\mathbf{Z})$ depends strictly on the uniform spherical measure of $\mathbf{U}$, completely independent of the radial distribution $R$.

Consequently:

• The standard one-sample Student's $t$-statistic $T = \frac{\bar{X} \sqrt{n}}{S}$ has an exact $t_{n-1}$ distribution under the null hypothesis $\mu = 0$ not only for Gaussian samples, but for any spherical population distribution!
• The ANOVA $F$-statistic has an exact $F$ distribution under the null hypothesis for all spherical distributions!

This invariance provides deep mathematical insight into the extraordinary empirical robustness of Student's $t$-tests and $F$-tests against departures from normality: whenever the data-generating mechanism preserves spherical symmetry, the exact null distribution of the test statistic is completely unchanged.