Exact & Asymptotic Inference in Linear Models: General Linear Hypotheses, Confidence Ellipsoids & Simultaneous Intervals

In empirical research, statistical hypotheses rarely take the elementary form of testing whether a single isolated parameter vanishes in a vacuum. Econometricians testing constant returns to scale in a Cobb–Douglas production function $\ln(Y_i) = \beta_0 + \beta_1 \ln(K_i) + \beta_2 \ln(L_i) + \varepsilon_i$ must test whether the capital and labor elasticities sum to unity ($\beta_1 + \beta_2 = 1$). Biostatisticians comparing four clinical treatment regimens must test whether three differential treatment effects are simultaneously zero ($\beta_1 = \beta_2 = \beta_3 = 0$). Agricultural scientists evaluating fertilizer formulations must test whether nitrogen and phosphorus have identical marginal yields ($\beta_1 - \beta_2 = 0$) while simultaneously testing whether a potassium interaction vanishes ($\beta_3 = 0$). Every one of these diverse scientific questions is a special case of the **General Linear Hypothesis (GLH)** on the $p \times 1$ parameter vector $\boldsymbol{\beta}$ of the classical normal linear model $\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$ with $\boldsymbol{\varepsilon} \sim \mathcal{N}_n(\mathbf{0}_n, \sigma^2 \mathbf{I}_n)$ and $\text{rank}(\mathbf{X}) = p$: $$H_0: \mathbf{C}\boldsymbol{\beta} = \mathbf{d} \qquad \text{versus} \qquad H_1: \mathbf{C}\boldsymbol{\beta} \neq \mathbf{d}$$ Here, $\mathbf{C} \in \mathbb{R}^{q \times p}$ is a known, deterministic **hypothesis (or coefficient) matrix** specifying $q$ linear combinations of the parameters, and $\mathbf{d} \in \mathbb{R}^{q \times 1}$ is a known vector of hypothesized constants. When $\mathbf{d} = \mathbf{0}_q$, the null hypothesis is termed a **homogeneous linear hypothesis**. To ensure that the $q$ scalar equations inside $\mathbf{C}\boldsymbol{\beta} = \mathbf{d}$ are algebraically consistent and contain no redundant linear dependencies, we require that $\mathbf{C}$ possess **full row rank**: $$\text{rank}(\mathbf{C}) = q \le p$$ If a researcher initially writes down a set of linear equations where one equation is a linear combination of the others, redundant rows are simply eliminated until the remaining $q$ rows are linearly independent in $\mathbb{R}^p$. Geometrically, imposing the $q$ linearly independent restrictions $\mathbf{C}\boldsymbol{\beta} = \mathbf{d}$ constrains the parameter vector $\boldsymbol{\beta}$ to lie within an affine subspace (flat manifold) of dimension $p - q$ embedded inside $\mathbb{R}^p$. To construct an exact test statistic from the unrestricted Ordinary Least Squares estimator $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$, we examine the **discrepancy vector** $\mathbf{w} = \mathbf{C}\hat{\boldsymbol{\beta}} - \mathbf{d} \in \mathbb{R}^{q \times 1}$. Because $\hat{\boldsymbol{\beta}} \sim \mathcal{N}_p(\boldsymbol{\beta}, \sigma^2(\mathbf{X}^T\mathbf{X})^{-1})$, any affine transformation of $\hat{\boldsymbol{\beta}}$ follows a multivariate normal distribution: $$\mathbf{C}\hat{\boldsymbol{\beta}} - \mathbf{d} \sim \mathcal{N}_q \left( \mathbf{C}\boldsymbol{\beta} - \mathbf{d}, \; \sigma^2 \mathbf{C}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{C}^T \right)$$ Because $\text{rank}(\mathbf{C}) = q$ and $(\mathbf{X}^T\mathbf{X})^{-1}$ is strictly positive definite, the $q \times q$ symmetric matrix $\mathbf{G} = \mathbf{C}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{C}^T$ is strictly positive definite and invertible. Consequently, standardizing the discrepancy vector by its inverse covariance matrix yields a quadratic form that follows a non-central chi-square distribution with $q$ degrees of freedom and non-centrality parameter $\lambda = (\mathbf{C}\boldsymbol{\beta} - \mathbf{d})^T [\sigma^2 \mathbf{C}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{C}^T]^{-1} (\mathbf{C}\boldsymbol{\beta} - \mathbf{d})$: $$\frac{Q_H}{\sigma^2} = \frac{(\mathbf{C}\hat{\boldsymbol{\beta}} - \mathbf{d})^T \left[ \mathbf{C}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{C}^T \right]^{-1} (\mathbf{C}\hat{\boldsymbol{\beta}} - \mathbf{d})}{\sigma^2} \sim \chi_q^2(\lambda)$$ Under the null hypothesis $H_0: \mathbf{C}\boldsymbol{\beta} = \mathbf{d}$, the non-centrality parameter vanishes ($\lambda = 0$), so $Q_H / \sigma^2 \sim \chi_q^2$ follows a central chi-square distribution with $q$ degrees of freedom. Because $\hat{\boldsymbol{\beta}}$ (and hence $Q_H$) is statistically independent of the residual sum of squares $\text{SSE} = (n - p)s^2$ where $\text{SSE}/\sigma^2 \sim \chi_{n-p}^2$, dividing the mean square $(Q_H / q)$ by $s^2 = \text{SSE}/(n - p)$ cancels the unknown scale parameter $\sigma^2$ and produces the exact **General Linear Hypothesis $F$-Statistic**: $$F = \frac{(\mathbf{C}\hat{\boldsymbol{\beta}} - \mathbf{d})^T \left[ \mathbf{C}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{C}^T \right]^{-1} (\mathbf{C}\hat{\boldsymbol{\beta}} - \mathbf{d}) / q}{\text{SSE} / (n - p)} = \frac{Q_H / q}{s^2} \sim F_{q, \, n-p} \quad \text{under } H_0$$ When $H_0$ is false ($\mathbf{C}\boldsymbol{\beta} \neq \mathbf{d}$), $F$ follows a non-central $F_{q, n-p}(\lambda)$ distribution whose expectation exceeds unity, so we reject $H_0$ at significance level $\alpha$ whenever $F > F_{\alpha, \, q, \, n-p}$. ### Algebraic Invariance of the General Linear Hypothesis F-Statistic Under Equivalent Contrast Matrices A crucial theoretical property of the General Linear Hypothesis test is that the value of the $F$-statistic depends only on the linear subspace (or affine restriction manifold) defined by the null hypothesis, and not on the particular algebraic basis chosen to write the contrast matrix. Suppose two statisticians formulate the same structural hypothesis—for instance, the simultaneous equality of three slope coefficients—using two different full-row-rank contrast matrices that are related to one another by premultiplication by an invertible $q \times q$ transformation matrix. When the transformed contrast matrix and transformed right-hand-side vector are substituted into the Wald quadratic form, the invertible transformation matrix and its transpose factor out of the middle inverse matrix and cancel identically with the outer factors. Thus, any two algebraically equivalent formulations of a linear restriction yield the exact same numerator sum of squares, degrees of freedom, and $p$-value.

A profound theorem of linear model theory states that the Wald-type quadratic form $Q_H = (\mathbf{C}\hat{\boldsymbol{\beta}} - \mathbf{d})^T [\mathbf{C}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{C}^T]^{-1} (\mathbf{C}\hat{\boldsymbol{\beta}} - \mathbf{d})$ derived in Section 3.1 is identically equal to the drop in the Residual Sum of Squares between the constrained (Reduced) model where $H_0: \mathbf{C}\boldsymbol{\beta} = \mathbf{d}$ is enforced and the unconstrained (Full) model! To prove this equivalence, we derive the Restricted Least Squares (RLS) estimator $\hat{\boldsymbol{\beta}}_R$, which minimizes $\|\mathbf{Y} - \mathbf{X}\mathbf{b}\|_2^2$ subject to the exact linear equality constraint $\mathbf{C}\mathbf{b} = \mathbf{d}$.

Formulate the Lagrangian objective function with a $q \times 1$ vector of Lagrange multipliers $2\boldsymbol{\lambda} \in \mathbb{R}^q$:

Setting the partial derivatives with respect to $\mathbf{b}$ and $\boldsymbol{\lambda}$ equal to zero at the constrained minimizer $(\hat{\boldsymbol{\beta}}_R, \hat{\boldsymbol{\lambda}})$ gives the first-order Karush–Kuhn–Tucker (KKT) optimality system:

Premultiplying the first equation by $(\mathbf{X}^T\mathbf{X})^{-1}$ and recognizing the unrestricted OLS estimator $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$ yields:

To solve for the Lagrange multiplier vector $\hat{\boldsymbol{\lambda}}$, premultiply both sides of this expression by $\mathbf{C}$ and invoke the second KKT constraint $\mathbf{C}\hat{\boldsymbol{\beta}}_R = \mathbf{d}$:

Substituting $\hat{\boldsymbol{\lambda}}$ back into the equation for $\hat{\boldsymbol{\beta}}_R$ provides the closed-form Restricted Least Squares (RLS) Estimator:

Notice how transparent this formula is: if the unrestricted OLS estimator already satisfies the null hypothesis ($\mathbf{C}\hat{\boldsymbol{\beta}} = \mathbf{d}$), the correction term vanishes and $\hat{\boldsymbol{\beta}}_R = \hat{\boldsymbol{\beta}}$. Otherwise, $\hat{\boldsymbol{\beta}}_R$ projects $\hat{\boldsymbol{\beta}}$ obliquely onto the constraint manifold $\{\mathbf{b} : \mathbf{C}\mathbf{b} = \mathbf{d}\}$ in the metric of $(\mathbf{X}^T\mathbf{X})$. Under $H_0: \mathbf{C}\boldsymbol{\beta} = \mathbf{d}$, $\hat{\boldsymbol{\beta}}_R$ is strictly unbiased ($\mathbb{E}[\hat{\boldsymbol{\beta}}_R] = \boldsymbol{\beta}$) and has a smaller covariance matrix than $\hat{\boldsymbol{\beta}}$ in the Loewner sense:

Now let us decompose the residual vector of the Restricted (Reduced) model, $\mathbf{e}_R = \mathbf{Y} - \mathbf{X}\hat{\boldsymbol{\beta}}_R$, by adding and subtracting the unrestricted fitted vector $\hat{\mathbf{Y}} = \mathbf{X}\hat{\boldsymbol{\beta}}$:

where $\mathbf{e}_F = \mathbf{Y} - \mathbf{X}\hat{\boldsymbol{\beta}}$ is the residual vector of the unrestricted Full model. Because $\mathbf{X}^T \mathbf{e}_F = \mathbf{0}_p$ by the normal equations of the full model, the two vectors $\mathbf{e}_F \in \mathcal{C}(\mathbf{X})^\perp$ and $\mathbf{X}(\hat{\boldsymbol{\beta}} - \hat{\boldsymbol{\beta}}_R) \in \mathcal{C}(\mathbf{X})$ are strictly orthogonal! Taking squared Euclidean norms yields the Pythagorean identity:

Substituting $\hat{\boldsymbol{\beta}} - \hat{\boldsymbol{\beta}}_R = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{C}^T [\mathbf{C}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{C}^T]^{-1} (\mathbf{C}\hat{\boldsymbol{\beta}} - \mathbf{d})$ into the quadratic term simplifies immediately to:

This establishes the celebrated Full-versus-Reduced Model General Linear $F$-Test:

where the last equality holds whenever both the Full and Reduced models share the same dependent variable $Y$ and include an intercept so they share the same Total Sum of Squares $\text{SST}$. Moreover, under Gaussian errors, the Gaussian likelihood ratio statistic $\Lambda = \sup_{H_0} L(\boldsymbol{\beta}, \sigma^2) / \sup_{\Theta} L(\boldsymbol{\beta}, \sigma^2)$ equals $(\text{SSE}_F / \text{SSE}_R)^{n/2} = (1 + \frac{q}{n-p} F)^{-n/2}$, which is a strictly decreasing function of $F$, proving that the General Linear $F$-test is identically the exact Likelihood Ratio Test (LRT)!

Bias-Variance Trade-Off and Gauss-Markov Optimality of Restricted Least Squares

The Restricted Least Squares estimator derived via Lagrange multipliers illuminates the fundamental interplay between structural prior information and sampling precision. When the linear restriction imposed by the null hypothesis holds exactly in the underlying population, the Restricted Least Squares estimator is unbiased and achieves a strictly smaller variance-covariance matrix (in the Loewner positive semi-definite ordering) than the unrestricted Ordinary Least Squares estimator, because projecting the estimate onto the lower-dimensional constraint manifold eliminates $q$ dimensions of orthogonal estimation noise. However, if the imposed restriction is false in the population, the Restricted Least Squares estimator incurs a systematic specification bias proportional to the violation of the constraint, and the numerator of the General Linear $F$-statistic compares the squared magnitude of this restriction violation against the baseline error variance.

Let us specialize the General Linear Hypothesis to the most frequently encountered single-degree-of-freedom ($q = 1$) and multi-degree-of-freedom hypotheses in applied regression analysis.

1. Testing a Single Linear Combination $H_0: \mathbf{c}^T \boldsymbol{\beta} = d$ ($q = 1$)

Let $\mathbf{c} = (c_0, c_1, \dots, c_k)^T \in \mathbb{R}^p \setminus \{\mathbf{0}\}$ be a single column vector defining a scalar parametric target $\theta = \mathbf{c}^T \boldsymbol{\beta} = \sum_{j=0}^k c_j \beta_j$. By the Matrix Gauss–Markov Theorem, its unique BLUE is $\hat{\theta} = \mathbf{c}^T \hat{\boldsymbol{\beta}}$, which under normal errors follows:

Replacing $\sigma^2$ by $s^2 = \text{MSE} = \text{SSE}/(n - p)$ gives the estimated standard error of $\mathbf{c}^T \hat{\boldsymbol{\beta}}$:

To test $H_0: \mathbf{c}^T \boldsymbol{\beta} = d$ (against a two-sided alternative $H_1: \mathbf{c}^T \boldsymbol{\beta} \neq d$, or a one-sided alternative $H_1: \mathbf{c}^T \boldsymbol{\beta} > d$ or @@MATH_TOKEN_3@@lt; d$), we use the exact Student $t$-statistic with $n - p$ degrees of freedom:

Notice that squaring $t$ reproduces the $q = 1$ General Linear $F$-statistic: $t^2 = \frac{(\mathbf{c}^T\hat{\boldsymbol{\beta}} - d)^2}{s^2 \mathbf{c}^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{c}} = F \sim F_{1, n-p}$. The corresponding exact $100(1 - \alpha)\%$ confidence interval for $\mathbf{c}^T \boldsymbol{\beta}$ is $\mathbf{c}^T \hat{\boldsymbol{\beta}} \pm t_{\alpha/2, n-p} \, s \sqrt{\mathbf{c}^T (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{c}}$.

2. Individual Partial $t$-Tests $H_0: \beta_j = \beta_{j,0}$

Choosing $\mathbf{c} = \mathbf{e}_j$ (the $j$-th standard basis vector with $1$ in position $j$ and $0$ elsewhere) selects a single regression coefficient $\beta_j$. Then $\mathbf{c}^T (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{c} = c_{jj}$, the $j$-th diagonal entry of $(\mathbf{X}^T\mathbf{X})^{-1}$, so $\text{SE}(\hat{\beta}_j) = s \sqrt{c_{jj}}$ and:

When $\beta_{j,0} = 0$, $t_j^2 = F_j = \frac{\text{SSR}(X_j \mid \mathbf{X}_{(-j)})}{\text{MSE}}$ is the marginal (Type III) partial $F$-test for removing predictor $X_j$ from the full model.

3. Testing Equality of Two Slope Parameters $H_0: \beta_j = \beta_m$

To test whether two predictors $X_j$ and $X_m$ (measured in commensurate units) exert identical marginal effects on $Y$, we rewrite $H_0: \beta_j - \beta_m = 0$ by setting $\mathbf{c} = \mathbf{e}_j - \mathbf{e}_m$. Evaluating the quadratic form $\mathbf{c}^T (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{c} = c_{jj} + c_{mm} - 2 c_{jm}$ yields the standard error of the difference between two estimated slopes:

Alternatively, we can form the Reduced model directly by substituting $\beta_j = \beta_m$ into the regression equation:

which collapses the two separate predictors $x_{ij}$ and $x_{im}$ into a single composite sum variable $z_i = x_{ij} + x_{im}$, reducing the number of parameters by $q = 1$, and then compute $F = (\text{SSE}_R - \text{SSE}_F) / \text{MSE}_F = t^2$.

4. Testing Linear Structural Restrictions via Reparameterization

Similarly, to test $H_0: \beta_1 + \beta_2 = 1$ in $Y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \varepsilon_i$ via the Reduced-model method, substitute $\beta_2 = 1 - \beta_1$ into the model:

Thus, regressing the transformed response $Y_i^* = Y_i - x_{i2}$ on the single transformed regressor $x_i^* = x_{i1} - x_{i2}$ yields the exact Restricted Least Squares sum of squares $\text{SSE}_R$ with $n - 2$ degrees of freedom!

Commensurability of Units When Testing Equality or Sums of Regression Coefficients

When formulating linear combinations and equality tests across multiple regression slopes, applied statisticians must verify that the underlying variables are measured in commensurate physical or economic units. Testing whether a single partial regression coefficient equals zero is invariant to rescaling the predictor: multiplying a predictor by a thousand divides its estimated slope and its standard error by a thousand, leaving the individual $t$-statistic unchanged. In sharp contrast, testing whether two slope coefficients are equal to one another ($H_0: \beta_1 - \beta_2 = 0$) is meaningful only when both predictors share a common scale—such as dollars of expenditure on two advertising channels, percentage shares, or dimensionless logarithmic elasticities in a Cobb–Douglas function. If two predictors are measured in incompatible physical units, testing equality of their raw slopes is arbitrary unless the variables are first standardized to dimensionless z-scores.

When an investigator wishes to make a joint probability statement about the entire $p \times 1$ parameter vector $\boldsymbol{\beta}$ (or a $q \times 1$ subvector $\boldsymbol{\beta}_S = \mathbf{C}\boldsymbol{\beta}$), substituting $\mathbf{C} = \mathbf{I}_p$ and $\mathbf{d} = \boldsymbol{\beta}$ into the General Linear Hypothesis pivotal quantity shows that:

Inverting this pivotal quantity defines the exact $100(1 - \alpha)\%$ Joint Confidence Region for $\boldsymbol{\beta}$:

Because $\mathbf{X}^T\mathbf{X}$ is symmetric and strictly positive definite, the inequality $(\boldsymbol{\beta} - \hat{\boldsymbol{\beta}})^T (\mathbf{X}^T\mathbf{X}) (\boldsymbol{\beta} - \hat{\boldsymbol{\beta}}) \le c^2$ describes the interior and boundary of a solid $p$-dimensional hyper-ellipsoid centered at the OLS estimate $\hat{\boldsymbol{\beta}}$. More generally, for any $q \times 1$ parameter subvector $\boldsymbol{\theta} = \mathbf{C}\boldsymbol{\beta}$ (such as the two slopes $(\beta_1, \beta_2)^T$ with $q = 2$), the exact $100(1 - \alpha)\%$ joint confidence ellipsoid is:

Geometric Orientation of the Confidence Ellipsoid vs the Marginal Rectangle

Why is a joint confidence ellipsoid essential when predictors $X_1$ and $X_2$ are correlated? Suppose we construct separate $100(1 - \alpha)\%$ marginal $t$-intervals $\hat{\beta}_1 \pm t_{\alpha/2, n-p} \text{SE}(\hat{\beta}_1)$ and $\hat{\beta}_2 \pm t_{\alpha/2, n-p} \text{SE}(\hat{\beta}_2)$. Taking their Cartesian product forms an axis-aligned rectangular box in the $(\beta_1, \beta_2)$-plane. This naive rectangle fails in two ways:

1. Distorted Joint Coverage Probability: Even if $\hat{\beta}_1$ and $\hat{\beta}_2$ were independent, the probability that both independent $95\%$ intervals simultaneously cover their targets would be $(0.95)^2 = 0.9025 < 0.95$.

2. Tilt Induced by Predictor Collinearity: Recall from Problem 2.3 that when $X_1$ and $X_2$ have positive sample correlation $r_{12} > 0$, the off-diagonal element of $(\mathbf{X}_c^T\mathbf{X}_c)^{-1}$ is negative, meaning $\text{Cov}(\hat{\beta}_1, \hat{\beta}_2) < 0$ (and $\rho(\hat{\beta}_1, \hat{\beta}_2) = -r_{12}$). Intuitively, if $X_1$ and $X_2$ move together in the sample, overestimating $\beta_1$ can be compensated by underestimating $\beta_2$ while keeping the fitted predictions $\hat{\beta}_1 x_{i1} + \hat{\beta}_2 x_{i2}$ nearly unchanged! Conversely, the quadratic form matrix $\mathbf{X}_c^T\mathbf{X}_c$ has positive off-diagonal entry $S_{12} > 0$, so the confidence ellipse:

is tilted along a negative $45^\circ$-like principal axis (running from top-left to bottom-right). The eigenvalues $\lambda_{\min}, \lambda_{\max}$ of $\mathbf{X}_c^T\mathbf{X}_c$ govern the semi-axis lengths of the ellipse: the major (longest) semi-axis has length $\sqrt{2 s^2 F_{\alpha, 2, n-p} / \lambda_{\min}}$ along the direction of near-collinearity, while the minor (shortest) semi-axis has length $\sqrt{2 s^2 F_{\alpha, 2, n-p} / \lambda_{\max}}$. Consequently, a parameter pair $(\beta_1, \beta_2)$ (such as $(0, 0)$) can lie well outside the tilted joint confidence ellipse—so the joint $F$-test rejects $H_0: \beta_1 = \beta_2 = 0$—even when $0$ lies inside both individual marginal $t$-intervals! Conversely, corners of the marginal rectangle where $\beta_1$ and $\beta_2$ deviate in the same direction lie far outside the tilted ellipse.

Resolving Apparent Contradictions Between the Overall ANOVA F-Test and Individual t-Tests

In empirical regression modeling with correlated predictors, analysts frequently encounter the puzzling situation in which the overall Analysis of Variance $F$-test rejects the global null hypothesis with an extremely small $p$-value, yet every single individual partial $t$-test fails to reach statistical significance. The geometric analysis of joint confidence ellipsoids completely resolves this apparent paradox. Each individual $t$-test projects the multidimensional confidence ellipsoid orthogonally onto a single coordinate axis, measuring only what a predictor contributes uniquely after all other correlated predictors have already been accounted for. When two predictors are strongly correlated, neither variable possesses much unique variation orthogonal to its partner, so the axis projections of the tilted confidence ellipsoid are wide and span zero on each individual axis, even though the narrow tilted ellipsoid itself misses the origin by a wide margin along the diagonal direction of joint variation.

This geometric divergence between the joint confidence ellipsoid and the axis-aligned marginal confidence box underscores a cardinal rule of multiple regression modeling: an analyst must never drop two or more correlated predictors simultaneously from a regression equation simply because their individual partial $t$-statistics are both smaller than the critical threshold! Deleting both variables at once corresponds to testing the joint null hypothesis that both parameters equal zero, which requires evaluating whether the origin lies inside the joint two-dimensional confidence ellipse via the partial $F$-test. Frequently, dropping one of the two collinear predictors immediately causes the remaining predictor's standard error to collapse and its $t$-statistic to become highly significant.

Whenever an analyst constructs $m \ge 2$ confidence intervals or conducts $m$ hypothesis tests from the same regression dataset, the Familywise Error Rate (FWER)—defined as the probability that at least one confidence interval in the family fails to cover its true parameter—exceeds the nominal pointwise level $\alpha$. Let $E_j$ denote the error event that the $j$-th confidence interval $I_j$ fails to cover its true target $\theta_j = \mathbf{c}_j^T \boldsymbol{\beta}$ ($j = 1, \dots, m$). The joint (simultaneous) confidence level of the family $\{I_1, \dots, I_m\}$ is:

Three classical simultaneous inference procedures control the familywise error rate $\mathbb{P}(\bigcup_{j=1}^m E_j) \le \alpha$:

1. The Bonferroni Simultaneous Intervals (Finite Pre-Specified Family of Size $m$)

By Boole's Inequality (the finite union bound), for any events $E_1, \dots, E_m$ regardless of their dependence structure:

Therefore, if we construct each of the $m$ pre-selected confidence intervals at individual error rate $\alpha / m$ (so $\mathbb{P}(E_j) = \alpha / m$), the familywise error rate is guaranteed not to exceed $\sum_{j=1}^m (\alpha/m) = \alpha$:

A slightly sharper bound for normal linear models is the Šidák inequality (valid for multivariate Student-$t$ distributions with product-type correlations), which replaces $\alpha/(2m)$ by $\frac{1}{2}[1 - (1 - \alpha)^{1/m}]$, or the exact multivariate-$t$ Studentized Maximum Modulus distribution.

2. Scheffé's $S$-Method (All Infinitely Many Linear Combinations in a $q$-Dimensional Subspace)

What if the investigator does not fix a small finite number $m$ of contrasts in advance, or wishes to engage in post-hoc data snooping across all possible linear combinations $\mathbf{c}^T \boldsymbol{\beta}$ belonging to a $q$-dimensional subspace spanned by the rows of $\mathbf{C} \in \mathbb{R}^{q \times p}$? Henry Scheffé (1953) solved this problem by projecting the $q$-dimensional joint confidence ellipsoid $\mathcal{E}_{1-\alpha}(\mathbf{C}\boldsymbol{\beta})$ onto every one-dimensional line! By the Cauchy–Schwarz Inequality (in the metric of $\mathbf{G} = \mathbf{C}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{C}^T$), for any vector $\mathbf{a} \in \mathbb{R}^q \setminus \{\mathbf{0}\}$:

Taking square roots proves that with exact probability $1 - \alpha$, simultaneously for every $\mathbf{a} \in \mathbb{R}^q$ (where $\mathbf{c}^T = \mathbf{a}^T \mathbf{C}$):

When $\mathbf{C} = \mathbf{I}_p$ ($q = p$), $S_\alpha = \sqrt{p F_{\alpha, p, n-p}}$ provides simultaneous intervals for all $\mathbf{x}_0^T \boldsymbol{\beta}$ across the entire predictor space $\mathbf{x}_0 \in \mathbb{R}^p$ (the multiple-regression generalization of the Working–Hotelling band!). Moreover, Scheffé's method enjoys exact coherence with the $F$-test: the $F$-test rejects $H_0: \mathbf{C}\boldsymbol{\beta} = \mathbf{d}$ at level $\alpha$ if and only if there exists at least one linear combination $\mathbf{a}^T \mathbf{C}\boldsymbol{\beta}$ whose Scheffé interval excludes $\mathbf{a}^T \mathbf{d}$.

Strategic Choice Between Bonferroni and Scheffé:

Because both methods use the identical standard error $\text{SE}(\mathbf{c}_j^T\hat{\boldsymbol{\beta}})$, the analyst simply compares the two critical multipliers $B_\alpha = t_{\alpha/(2m), n-p}$ and $S_\alpha = \sqrt{q F_{\alpha, q, n-p}}$ before examining the data and selects whichever multiplier is smaller! For a small number of pre-planned comparisons ($m \approx q$ or $m$ slightly larger than $q$), the Bonferroni multiplier $t_{\alpha/(2m), n-p}$ is strictly smaller than $\sqrt{q F_{\alpha, q, n-p}}$, yielding narrower intervals; for large $m \gg q$ or post-hoc exploration, Scheffé's multiplier remains bounded at $\sqrt{q F_{\alpha, q, n-p}}$ whereas $t_{\alpha/(2m), n-p} \to \infty$.

Controlling Familywise Error Rates in Confirmatory Versus Exploratory Regression Studies

Modern reproducible science places heavy emphasis on controlling false-positive discovery rates when multiple inferences are drawn from a single regression model. In a confirmatory clinical trial or engineering validation study where a small number of specific linear contrasts are pre-registered in the study protocol prior to data inspection, the Bonferroni (or Holm–Bonferroni step-down) procedure provides sharp, easily computed simultaneous intervals with minimal loss of statistical power. Conversely, in exploratory observational studies where researchers examine numerous post-hoc subgroup comparisons, pairwise treatment differences, and multi-parameter contrasts suggested by the observed data themselves, pre-planned multiplicity adjustments are invalid because the effective number of implicit comparisons is unbounded. In such post-hoc settings, Scheffé's $S$-method provides an ironclad mathematical guarantee that the familywise error rate across all infinitely many linear combinations in the contrast subspace will never exceed the nominal significance level.

Extending Section 1.6 to the general $p$-parameter multiple regression model, suppose we specify a target covariate vector $\mathbf{x}_0 = (1, x_{01}, x_{02}, \dots, x_{0k})^T \in \mathbb{R}^p$. We distinguish once again between estimating the deterministic conditional mean response $\mu_{Y \mid \mathbf{x}_0} = \mathbb{E}[Y \mid \mathbf{x}_0] = \mathbf{x}_0^T \boldsymbol{\beta}$ and predicting a future random realization $Y_{\text{new}} = \mathbf{x}_0^T \boldsymbol{\beta} + \varepsilon_{\text{new}}$, where $\varepsilon_{\text{new}} \sim \mathcal{N}(0, \sigma^2)$ is independent of the historical training sample $\mathbf{Y}$.

The BLUE point estimator of $\mu_{Y \mid \mathbf{x}_0}$ (and minimum-MSE linear unbiased predictor of $Y_{\text{new}}$) is:

Define the out-of-sample leverage at $\mathbf{x}_0$ as the quadratic form $h_{00} = \mathbf{x}_0^T (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{x}_0$. The sampling variance of $\hat{Y}_0$ and the variance of the prediction error $Y_{\text{new}} - \hat{Y}_0$ are:

Consequently, the exact $100(1 - \alpha)\%$ Confidence Interval for $\mathbb{E}[Y \mid \mathbf{x}_0]$ and Prediction Interval for $Y_{\text{new}}$ are:

If we wish to construct simultaneous prediction intervals for $m$ independent future observations at $\mathbf{x}_{0(1)}, \dots, \mathbf{x}_{0(m)}$, we apply either the Bonferroni multiplier $t_{\alpha/(2m), n-p}$ or the Scheffé simultaneous prediction multiplier $\sqrt{m F_{\alpha, m, n-p}}$ to $s\sqrt{1 + \mathbf{x}_{0(j)}^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{x}_{0(j)}}$.

The Geometry of Hidden Extrapolation and the Minimum-Volume Regressor Ellipsoid

In simple linear regression ($k = 1$), detecting whether a target point $x_0$ represents an dangerous extrapolation is trivial: one simply checks whether $x_0$ lies outside the sample range $[x_{\min}, x_{\max}]$. In multiple linear regression ($k \ge 2$), checking whether each individual coordinate $x_{0j}$ falls inside its marginal interval $[x_{j, \min}, x_{j, \max}]$ is grossly insufficient! When predictors $X_1, \dots, X_k$ are correlated, the historical sample points occupy a narrow, tilted cigar-shaped region inside the $k$-dimensional hyper-rectangle $\prod_{j=1}^k [x_{j, \min}, x_{j, \max}]$. A target point $\mathbf{x}_0$ can have every coordinate well within its individual marginal range and yet lie completely outside the joint multidimensional cloud of observed data—a perilous phenomenon called hidden extrapolation.

To detect hidden extrapolation rigorously without relying on two-dimensional scatterplots, Weisberg and Montgomery define the Regressor Variable Hull (RVH) as the smallest convex hull or minimum-volume ellipsoid enclosing the $n$ observed sample points $\mathbf{x}_1, \dots, \mathbf{x}_n$. Because the diagonal elements of the Hat matrix are $h_{ii} = \mathbf{x}_i^T (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{x}_i$, let $h_{\max} = \max_{1 \le i \le n} h_{ii}$ be the maximum leverage observed in the training sample. The Regressor Variable Ellipsoid is defined as the set of all covariate vectors $\mathbf{x}_0$ whose leverage does not exceed $h_{\max}$:

Whenever a proposed forecast point $\mathbf{x}_0$ yields $h_{00} > h_{\max}$, $\mathbf{x}_0$ lies further from the multivariate sample centroid (in Mahalanobis distance) than any observation in the training dataset, signaling hidden extrapolation.

Asymptotic Inference Under Non-Normal Errors and Likelihood Ratio, Wald & Lagrange Multiplier Trinity

What happens to the $t$-tests, $F$-tests, and confidence intervals of Sections 3.1–3.6 when the classical Gauss–Markov assumptions A1–A5 hold ($\mathbb{E}[\boldsymbol{\varepsilon}] = \mathbf{0}_n$, $\text{Cov}(\boldsymbol{\varepsilon}) = \sigma^2 \mathbf{I}_n$), but Assumption A6 fails—meaning the error terms $\varepsilon_i$ are non-normal (for example, skewed exponential, heavy-tailed Student-$t$, or bounded uniform disturbances)?

In finite samples without normality, $\hat{\boldsymbol{\beta}}$ remains unbiased and BLUE by the Gauss–Markov Theorem, and $s^2 = \text{SSE}/(n-p)$ remains exactly unbiased for $\sigma^2$, but $\hat{\boldsymbol{\beta}}$ is no longer exactly multivariate normal and $\text{SSE}/\sigma^2$ is no longer exactly chi-square. However, as the sample size $n \to \infty$, large-sample asymptotic distribution theory rescues classical regression inference under mild regularity conditions on the sequence of design matrices $\mathbf{X}_n \in \mathbb{R}^{n \times p}$, known as the Grenander–Noether Conditions:

1. Asymptotic Stability of the Gram Matrix: As $n \to \infty$, the average cross-product matrix converges to a finite, symmetric, strictly positive definite $p \times p$ limit matrix $\mathbf{Q}$:

Note that this condition implies $(\mathbf{X}_n^T \mathbf{X}_n)^{-1} = \frac{1}{n} \mathbf{Q}_n^{-1} \to \mathbf{0}_{p \times p}$, so $\text{Cov}(\hat{\boldsymbol{\beta}}_n) = \sigma^2(\mathbf{X}_n^T\mathbf{X}_n)^{-1} \to \mathbf{0}_{p \times p}$. By Chebyshev's inequality, vanishing variance plus zero bias immediately guarantees that $\hat{\boldsymbol{\beta}}_n$ is weakly consistent: $\hat{\boldsymbol{\beta}}_n \xrightarrow{P} \boldsymbol{\beta}$.

2. Vanishing Maximum Leverage (Lindeberg Condition): No single observation dominates the sample asymptotically, meaning the maximum diagonal element of the Hat matrix converges to zero:

Under these conditions, if $\varepsilon_1, \dots, \varepsilon_n$ are i.i.d. with mean $0$ and finite variance $\sigma^2 \in (0, \infty)$, the Lindeberg–Feller Multivariate Central Limit Theorem applies to the scaled sum $\frac{1}{\sqrt{n}} \mathbf{X}_n^T \boldsymbol{\varepsilon} = \frac{1}{\sqrt{n}} \sum_{i=1}^n \mathbf{x}_i \varepsilon_i$, establishing asymptotic normality of the OLS estimator:

Simultaneously, by the Weak Law of Large Numbers, $s_n^2 = \frac{1}{n-p}\boldsymbol{\varepsilon}^T(\mathbf{I}_n - \mathbf{H}_n)\boldsymbol{\varepsilon} = \frac{n}{n-p}\left(\frac{1}{n}\boldsymbol{\varepsilon}^T\boldsymbol{\varepsilon} - \frac{1}{n}\boldsymbol{\varepsilon}^T\mathbf{H}_n\boldsymbol{\varepsilon}\right) \xrightarrow{P} \sigma^2$ is a consistent estimator of $\sigma^2$. By Slutsky's Theorem and the Continuous Mapping Theorem, under $H_0: \mathbf{C}\boldsymbol{\beta} = \mathbf{d}$:

and $F_n = W_n / q \xrightarrow{d} \chi_q^2 / q$. Notice that as the denominator degrees of freedom $n - p \to \infty$, the random variable $F_{q, n-p}$ converges in distribution to $\chi_q^2 / q$! Therefore, the classical $t$-tests and $F$-tests are asymptotically valid for arbitrary non-normal error distributions with finite variance. (Caution: single-observation prediction intervals $\hat{y}_0 \pm t_{\alpha/2} s \sqrt{1 + h_{00}}$ still rely on the normality of the individual future error $\varepsilon_{\text{new}}$, because a single error $\varepsilon_{\text{new}}$ is not averaged by a Central Limit Theorem!).

Finally, in large-sample econometrics, testing $H_0: \mathbf{C}\boldsymbol{\beta} = \mathbf{d}$ can be conducted via any of the Asymptotic Trinity of Tests:

1. Wald Statistic ($W$): Computed solely from the unrestricted model: $W = n (\text{SSE}_R - \text{SSE}_F) / \text{SSE}_F = \frac{n}{n-p} q F$.

2. Likelihood Ratio Statistic ($LR$): $LR = n \ln(\text{SSE}_R / \text{SSE}_F) = n \ln\!\left(1 + \frac{W}{n}\right)$.

3. Lagrange Multiplier / Score Statistic ($LM$): Computed solely from the restricted model: $LM = n (\text{SSE}_R - \text{SSE}_F) / \text{SSE}_R = \frac{W}{1 + W/n} = n R_{\text{aux}}^2$, where $R_{\text{aux}}^2$ is the uncentered $R^2$ from regressing the restricted residuals $\mathbf{e}_R$ on $\mathbf{X}$.

In every linear regression model of sample size $n$, these three statistics obey the exact deterministic algebraic inequality:

while all three converge in distribution to the identical $\chi_q^2$ limit under $H_0$ as $n \to \infty$, since $W - LR = O_p(n^{-1})$ and $LR - LM = O_p(n^{-1})$.

Asymptotic Robustness and the Role of Heteroscedasticity-Consistent Inference

The asymptotic convergence results grounded in the Grenander–Noether conditions and the Lindeberg–Feller Central Limit Theorem explain why classical regression inference remains the workhorse of applied econometrics and social science even when disturbances depart from exact Gaussianity. Provided the sample size is moderately large and no single observation exercises asymptotically dominant leverage, the sampling distribution of the Ordinary Least Squares estimator is closely approximated by a multivariate normal distribution regardless of skewness or kurtosis in the error distribution. Crucially, however, asymptotic validity of the standard $t$-tests and $F$-tests still requires that the error variance be constant across observations (homoscedasticity); when error variance varies systematically with the predictors, the classical covariance formula remains inconsistent even in infinitely large samples, motivating the Generalized Least Squares and Huber–Eicker–White sandwich estimators developed in Unit 5.