Asymptotic Properties of Maximum Likelihood Estimators

Asymptotic statistical theory relies on formal modes of convergence for sequences of random variables defined on a probability space $(\Omega, \mathcal{F}, P)$. Let $\{T_n\}_{n \ge 1}$ be a sequence of estimators (random variables) and let $\theta$ be a fixed constant or target parameter.

1. Convergence in Probability (Weak Consistency)

A sequence of estimators $T_n$ converges in probability to $\theta$, denoted $T_n \xrightarrow{P} \theta$, if for every $\epsilon > 0$:

An estimator satisfying $T_n \xrightarrow{P} \theta$ is said to be weakly consistent. Weak consistency ensures that in sufficiently large samples, the probability of any noticeable estimation error arbitrarily close to zero vanishes.

2. Almost Sure Convergence (Strong Consistency)

A sequence $T_n$ converges almost surely (with probability $1$) to $\theta$, denoted $T_n \xrightarrow{\text{a.s.}} \theta$, if:

Equivalently, for every $\epsilon > 0$, $\lim_{N \to \infty} \mathbb{P}\left( \bigcup_{n=N}^\infty \{|T_n - \theta| \ge \epsilon\} \right) = 0$. An estimator satisfying $T_n \xrightarrow{\text{a.s.}} \theta$ is strongly consistent. Strong consistency implies weak consistency by the dominated convergence theorem.

3. Convergence in Mean Square ($L^2$ Consistency)

A sequence $T_n$ converges in mean square to $\theta$, denoted $T_n \xrightarrow{L^2} \theta$, if:

Because $\mathbb{E}[(T_n - \theta)^2] = \text{MSE}(T_n) = \text{Var}(T_n) + [\text{Bias}(T_n)]^2$, an estimator is $L^2$-consistent if and only if its variance and bias simultaneously vanish as $n \to \infty$. By Chebyshev's Inequality, $L^2$ convergence implies convergence in probability.

4. Convergence in Distribution (Weak Convergence)

A sequence of random variables $T_n$ with cumulative distribution functions $F_n(t) = \mathbb{P}(T_n \le t)$ converges in distribution to a limiting random variable $Z$ with CDF $F(t)$, denoted $T_n \xrightarrow{d} Z$, if:

for all continuity points $t$ of $F$. In statistical estimation, we typically analyze the normalized estimation error $\sqrt{n}(T_n - \theta)$ and prove that it converges in distribution to a Gaussian law $\mathcal{N}(0, V)$.

Under what conditions is the Maximum Likelihood Estimator guaranteed to converge to the true parameter $\theta_0$? We distinguish between local consistency (Cramér, 1946) and global consistency (Wald, 1949).

Cramér's Local Consistency Theorem

Let $X_1, X_2, \dots, X_n$ be an i.i.d. sample from a density $f(x; \theta)$ with true parameter $\theta_0$ residing in an open interval $\Theta \subseteq \mathbb{R}$. Assume the Cramér regularity conditions (C1)-(C6) hold:

1. The support $\mathcal{S} = \{x : f(x; \theta) > 0\}$ is independent of $\theta$.
2. For almost all $x$, $\ln f(x; \theta)$ is three times continuously differentiable with respect to $\theta$ on an open neighborhood of $\theta_0$.
3. For all $\theta \in \Theta$, the expected Fisher Information $I_1(\theta) = \mathbb{E}_\theta\left[ \left( \frac{\partial \ln f}{\partial \theta} \right)^2 \right]$ is finite and strictly positive ($0 < I_1(\theta) < \infty$).
4. There exists a measurable function $M(x)$ and an $\epsilon > 0$ such that for all $\theta \in (\theta_0 - \epsilon, \theta_0 + \epsilon)$:

Theorem: Existence of a Consistent Root

Under regularity conditions (1)-(4), with probability approaching $1$ as $n \to \infty$, the likelihood equation $\frac{\partial \ell_n(\theta)}{\partial \theta} = 0$ has a root $\hat{\theta}_n$ such that:

Proof:

Let $\epsilon > 0$ be chosen sufficiently small such that $(\theta_0 - \epsilon, \theta_0 + \epsilon) \subset \Theta$. Consider the normalized log-likelihood difference:

By the Strong Law of Large Numbers, as $n \to \infty$:

where $\text{KL}(f_0 \| f)$ is the Kullback–Leibler divergence. By Gibbs' Inequality (information inequality), the Kullback–Leibler divergence is strictly positive whenever the measures are distinct:

Thus, for any fixed $\theta \ne \theta_0$, the expectation is strictly negative! In particular, evaluate this at the two boundary points $\theta_1 = \theta_0 - \epsilon$ and $\theta_2 = \theta_0 + \epsilon$:

Because $\ell_n(\theta)$ is continuous, whenever $\ell_n(\theta_0 - \epsilon) < \ell_n(\theta_0)$ and $\ell_n(\theta_0 + \epsilon) < \ell_n(\theta_0)$, the function $\ell_n(\theta)$ must achieve a local maximum at some interior point $\hat{\theta}_n \in (\theta_0 - \epsilon, \theta_0 + \epsilon)$. At this local maximum, the derivative vanishes: $\ell_n'(\hat{\theta}_n) = 0$. Since this event occurs with probability approaching $1$ for any arbitrarily small $\epsilon > 0$, we have proved the existence of a consistent local root: $\hat{\theta}_n \xrightarrow{P} \theta_0$.

Wald's Theorem on Global Strong Consistency

Abraham Wald (1949) proved that if the parameter space $\Theta$ is compact (or compactifiable) and the log-density satisfies a uniform integrability condition $\mathbb{E}[\sup_{\theta \in U} \ln f(X; \theta)] < \infty$, then the global Maximum Likelihood Estimator is strongly consistent:

Wald's proof demonstrates that the empirical log-likelihood function $\frac{1}{n} \ell_n(\theta)$ converges uniformly across $\Theta$ to the deterministic limit $\mathbb{E}_{\theta_0}[\ln f(X; \theta)]$, which has a unique global maximum at $\theta = \theta_0$ by the strict negativity of the Kullback–Leibler divergence.

We now establish the centerpiece of large-sample likelihood theory: Maximum Likelihood Estimators converge at a $\sqrt{n}$ rate to a centered Gaussian distribution with variance equal to the inverse of the Fisher Information.

Theorem: Asymptotic Normality of the MLE

Under Cramér regularity conditions (C1)-(C6), any consistent root $\hat{\theta}_n$ of the likelihood equations satisfies:

where $I_1(\theta_0) = \mathbb{E}_{\theta_0}\left[ \left( \frac{\partial \ln f(X; \theta_0)}{\partial \theta} \right)^2 \right]$ is the single-observation Fisher Information.

Rigorous Proof:

By definition, $\hat{\theta}_n$ satisfies the likelihood equation:

Perform a first-order Taylor expansion of the score function $\ell_n'(\hat{\theta}_n)$ around the true parameter $\theta_0$:

where $\theta_n^*$ is an intermediate point lying on the line segment connecting $\hat{\theta}_n$ and $\theta_0$: $|\theta_n^* - \theta_0| \le |\hat{\theta}_n - \theta_0|$. Rearranging this linear equation to isolate the estimation error:

Multiplying both sides by $\sqrt{n}$:

Now analyze the asymptotic behavior of the numerator and denominator separately:

1. Analysis of the Numerator (Central Limit Theorem): The numerator is:

Each term $S_1(\theta_0; X_i)$ is an independent, identically distributed random variable. By Bartlett's First Identity:

The variance of each score term is the Fisher Information:

Because $0 < I_1(\theta_0) < \infty$, the classical Lindeberg–Lévy Central Limit Theorem applies directly:

2. Analysis of the Denominator (Law of Large Numbers and Remainder Domination): The denominator is:

Expand $\frac{\partial^2}{\partial \theta^2} \ln f(X_i; \theta_n^*)$ around $\theta_0$:

where $\tilde{\theta}_n$ lies between $\theta_n^*$ and $\theta_0$. Taking the average across the sample:

For the first term, by the Weak Law of Large Numbers:

by Bartlett's Second Identity. For the second term: Since $\hat{\theta}_n \xrightarrow{P} \theta_0$, we have $\theta_n^* \xrightarrow{P} \theta_0$, so $(\theta_n^* - \theta_0) \xrightarrow{P} 0$. By regularity condition (C6), $\left| \frac{1}{n} \sum_{i=1}^n \frac{\partial^3}{\partial \theta^3} \ln f(X_i; \tilde{\theta}_n) \right| \le \frac{1}{n} \sum_{i=1}^n M(X_i) \xrightarrow{P} \mathbb{E}[M(X)] < \infty$. The product of a term converging in probability to zero and a bounded term converges in probability to zero:

Therefore, by Slutsky's Theorem, the denominator converges in probability to the deterministic constant $I_1(\theta_0)$:

3. Combining via Slutsky's Theorem: We now have:

• Numerator $U_n = \frac{1}{\sqrt{n}} \ell_n'(\theta_0) \xrightarrow{d} U \sim \mathcal{N}(0, I_1(\theta_0))$
• Denominator $V_n = -\frac{1}{n} \ell_n''(\theta_n^*) \xrightarrow{P} c = I_1(\theta_0) > 0$

By Slutsky's Theorem (ratio of a weakly convergent sequence and a probability limit):

This completes the rigorous proof of asymptotic normality!

Multiparameter Generalization

For a $k$-dimensional parameter vector $\boldsymbol{\theta} \in \Theta \subseteq \mathbb{R}^k$, Taylor expansion of the score vector $\mathbf{S}_n(\hat{\boldsymbol{\theta}}) = \mathbf{0}$ yields:

where $\mathbf{I}_1(\boldsymbol{\theta}_0)$ is the $k \times k$ expected Fisher Information matrix per observation.

Bahadur Efficiency and Exact Slopes in Large Deviations

While Pitman asymptotic relative efficiency evaluates the local power of estimators in a $\mathcal{O}(1/\sqrt{n})$ neighborhood of the true parameter, R. R. Bahadur (1960) developed a global asymptotic efficiency theory based on large deviation principles. Let $T_n$ be a sequence of estimators of $\theta$, and consider the tail probability of estimation error exceeding a fixed non-zero threshold $\epsilon > 0$:

By Cramér's Large Deviation Theorem, this tail probability decays exponentially fast with sample size:

where $c(T, \epsilon) > 0$ is the Bahadur exact slope. Bahadur proved that for any regular estimator sequence, the exact slope is fundamentally bounded by the Kullback–Leibler divergence:

An estimator is defined to be Bahadur efficient if its exact slope achieves this theoretical upper bound for all $\epsilon > 0$. The Maximum Likelihood Estimator is uniquely Bahadur efficient! The exponential decay rate of large estimation errors is maximized by the MLE, reinforcing its optimality from both local Gaussian and global large-deviation perspectives.

A point estimator $T_n$ is said to be asymptotically efficient (or Best Asymptotically Normal / BAN) if:

Because the Cramér–Rao Lower Bound states that for any unbiased estimator $\text{Var}(T_n) \ge \frac{1}{n I_1(\theta)}$, an estimator achieving asymptotic variance $V(\theta) = \frac{1}{I_1(\theta)}$ attains the Cramér–Rao bound in the limit.

The Hájek–Le Cam Convolution Theorem

In 1970–1972, Jaroslav Hájek and Lucien Le Cam proved that the asymptotic variance $\frac{1}{I_1(\theta)}$ is indeed the fundamental lower bound on the asymptotic variance of all regular estimators.

Theorem (Hájek–Le Cam):

Let $\mathcal{P} = \{P_\theta : \theta \in \Theta\}$ be a locally asymptotically normal (LAN) family. Let $T_n$ be any regular estimator of $\theta$ (an estimator whose limiting distribution is locally stable under $\mathcal{O}(1/\sqrt{n})$ perturbations of $\theta$). Then the limiting distribution of $\sqrt{n}(T_n - \theta)$ is the convolution of a centered Gaussian distribution and an independent noise distribution:

where $Z_0 \sim \mathcal{N}\left(0, I_1(\theta)^{-1}\right)$ and $W$ is an independent random variable. Taking variances:

The limiting variance is strictly minimized if and only if $W \equiv 0$ almost surely, which occurs precisely for the Maximum Likelihood Estimator! This establishes that no regular estimator can ever beat the asymptotic precision of the MLE.

To construct asymptotic confidence intervals for $\theta$, we must estimate the asymptotic variance $V = \frac{1}{I_n(\theta)}$. Statisticians face a fundamental choice between two variance estimators:

1. Expected Information Estimator: $\hat{V}_{\text{exp}} = [\mathbf{I}_n(\hat{\boldsymbol{\theta}})]^{-1}$, where $\mathbf{I}_n(\boldsymbol{\theta}) = \mathbb{E}[-\nabla^2 \ell_n(\boldsymbol{\theta})]$.

2. Observed Information Estimator: $\hat{V}_{\text{obs}} = [J_n(\hat{\boldsymbol{\theta}})]^{-1}$, where $J_n(\hat{\boldsymbol{\theta}}) = -\nabla^2 \ell_n(\hat{\boldsymbol{\theta}})$.

In large samples, both estimators converge to the same limit: $\frac{1}{n} J_n(\hat{\theta}) \xrightarrow{P} I_1(\theta_0)$ and $\frac{1}{n} \mathbf{I}_n(\hat{\theta}) \xrightarrow{P} I_1(\theta_0)$. However, in finite samples, Bradley Efron and David Hinkley (1978) proved that observed information is superior to expected information.

The Efron–Hinkley Curvature Theorem

In translation and curved exponential models, the observed information $J_n$ can be decomposed as:

where $a(\mathbf{X})$ is an ancillary statistic measuring the statistical curvature of the realized likelihood function. Efron and Hinkley proved that the conditional distribution of the estimation error given the ancillary curvature:

converges to $\mathcal{N}(0, 1)$ with error rate $\mathcal{O}(1/n)$, whereas normalizing by expected information:

has a much coarser error rate of order $\mathcal{O}(1/\sqrt{n})$. Observed information measures the actual precision delivered by the realized data sample, whereas expected information averages over hypothetical samples that were never observed. Consequently, modern computational statistics universally prefers observed information $J_n(\hat{\boldsymbol{\theta}})^{-1}$ for standard error calculations.

In real-world applications, statistical models are idealizations. What happens to the Maximum Likelihood Estimator when the true data-generating distribution $G$ does not belong to the assumed parametric family $\mathcal{P} = \{F_\theta : \theta \in \Theta\}$?

Halbert White (1982) established the foundational theory of Quasi-Maximum Likelihood Estimation (QMLE).

The Pseudo-True Parameter

Let $g(x)$ denote the true, unknown probability density function generating the i.i.d. observations $X_1, \dots, X_n$. The Quasi-MLE $\hat{\theta}_n$ maximizes the assumed log-likelihood $\ell_n(\theta) = \sum_{i=1}^n \ln f(X_i; \theta)$. By the Strong Law of Large Numbers:

Notice that maximizing $\mathbb{E}_g[\ln f(X; \theta)]$ is mathematically identical to minimizing the Kullback–Leibler divergence between the true density $g$ and the assumed family $f(\cdot; \theta)$:

The pseudo-true parameter $\theta^*$ is defined as the parameter value that minimizes the information divergence:

White proved that under mild regularity conditions, the Quasi-MLE is strongly consistent for the pseudo-true parameter:

The MLE converges to the closest member of the candidate family to the true state of nature in the information-theoretic (Kullback–Leibler) projection sense.

Model Selection via Vuong's Likelihood Ratio Test for Misspecified Models

When an analyst compares two competing non-nested parametric families $\mathcal{F} = \{f(x; \boldsymbol{\beta})\}$ and $\mathcal{G} = \{g(x; \boldsymbol{\gamma})\}$, neither of which may be the true data-generating distribution $h_0(x)$, Quang Vuong (1989) formulated an asymptotic test based on the difference in Kullback–Leibler information criteria. Let $\ell_f(\hat{\boldsymbol{\beta}})$ and $\ell_g(\hat{\boldsymbol{\gamma}})$ be the maximized log-likelihoods under the two candidate models. The log-likelihood ratio statistic is:

Define the sample variance of the individual log-likelihood ratio contributions:

Vuong established that under the null hypothesis that both models are equally close to the true distribution in Kullback–Leibler distance ($H_0: \text{KL}(h_0 \| f) = \text{KL}(h_0 \| g)$):

If $V_n > +z_{1-\alpha/2}$, model $\mathcal{F}$ is statistically significantly closer to the true distribution; if $V_n < -z_{1-\alpha/2}$, model $\mathcal{G}$ is preferred. Vuong's test provides an asymptotically rigorous mechanism for discriminating between competing scientific theories under inevitable model misspecification.

Under model misspecification, Bartlett's Second Identity collapses! That is:

The expected Hessian matrix $\mathbf{A}(\theta^*)$ (the "bread") does not equal the outer product of the score vectors $\mathbf{B}(\theta^*)$ (the "meat").

Theorem: Asymptotic Distribution of QMLE

Applying a Taylor expansion to the misspecified score equation $\nabla \ell_n(\hat{\boldsymbol{\theta}}_n) = \mathbf{0}$:

where the asymptotic covariance matrix is the celebrated Huber–White Sandwich Matrix:

Empirical Estimation of the Sandwich Matrix

In practice, the sandwich covariance matrix is estimated directly from the sample without knowing the true density $g$:

1. The Hessian (bread) is estimated by the observed Hessian:

1. The score covariance (meat) is estimated by the empirical outer product of gradients:

1. The Heteroskedasticity-Consistent (Robust) Covariance Estimator is:

If the model is correctly specified, then by Bartlett's identity $\mathbf{A} = \mathbf{B}$, and the sandwich formula simplifies to $\mathbf{A}^{-1} \mathbf{A} \mathbf{A}^{-1} = \mathbf{A}^{-1} = \mathbf{I}_1(\theta)^{-1}$. The sandwich estimator provides valid standard errors and confidence intervals even when variance assumptions (e.g., homoskedasticity or equi-dispersion) are violated.

Estimating Functions and the Godambe Information Matrix

In semiparametric inference and econometrics, parameter estimators are frequently defined as solutions to an unbiased estimating equation:

In 1960, V. P. Godambe formulated a general optimality theory for estimating functions. Define:

• The sensitivity matrix: $\mathbf{S}_\psi = \mathbb{E}\left[ \nabla_{\boldsymbol{\theta}} \boldsymbol{\psi}(X; \boldsymbol{\theta}) \right]$
• The variability matrix: $\mathbf{V}_\psi = \mathbb{E}\left[ \boldsymbol{\psi}(X; \boldsymbol{\theta}) \boldsymbol{\psi}(X; \boldsymbol{\theta})^T \right]$

The Godambe Information Matrix is defined as:

Godambe proved that the asymptotic covariance of the estimator $\hat{\boldsymbol{\theta}}$ satisfies:

Notice that this is identical to the Huber–White sandwich covariance! Godambe proved that among all regular unbiased estimating functions, the score function $\boldsymbol{\psi}^* = \nabla \ln f(X; \boldsymbol{\theta})$ maximizes the Godambe Information matrix in the Loewner ordering, attaining the classical Fisher Information.

Estimating Functions and the Godambe Information Matrix

In semiparametric inference and econometrics, parameter estimators are frequently defined as solutions to an unbiased estimating equation:

In 1960, V. P. Godambe formulated a general optimality theory for estimating functions. Define:

• The sensitivity matrix: $\mathbf{S}_\psi = \mathbb{E}\left[ \nabla_{\boldsymbol{\theta}} \boldsymbol{\psi}(X; \boldsymbol{\theta}) \right]$
• The variability matrix: $\mathbf{V}_\psi = \mathbb{E}\left[ \boldsymbol{\psi}(X; \boldsymbol{\theta}) \boldsymbol{\psi}(X; \boldsymbol{\theta})^T \right]$

The Godambe Information Matrix is defined as:

Godambe proved that the asymptotic covariance of the estimator $\hat{\boldsymbol{\theta}}$ satisfies:

Notice that this is identical to the Huber–White sandwich covariance! Godambe proved that among all regular unbiased estimating functions, the score function $\boldsymbol{\psi}^* = \nabla \ln f(X; \boldsymbol{\theta})$ maximizes the Godambe Information matrix in the Loewner ordering, attaining the classical Fisher Information.

In parametric hypothesis testing and interval estimation for a null hypothesis $H_0: \boldsymbol{\theta} = \boldsymbol{\theta}_0$ versus $H_1: \boldsymbol{\theta} \ne \boldsymbol{\theta}_0$, three classical large-sample test statistics form the celebrated Likelihood Trinity:

1. The Wald Test ($W_n$):

Measures the squared distance between the unconstrained MLE $\hat{\boldsymbol{\theta}}$ and the hypothesized value $\boldsymbol{\theta}_0$, weighted by the Fisher Information curvature:

Computational Requirement: Requires fitting only the unrestricted model.

2. The Rao Score / Lagrange Multiplier Test ($R_n$):

Evaluates how close the score vector is to zero when evaluated at the restricted null parameter $\boldsymbol{\theta}_0$:

Computational Requirement: Requires fitting only the restricted model under $H_0$, avoiding unconstrained numerical optimization entirely.

3. The Likelihood Ratio Test (Wilks' Test, $\Lambda_n$):

Measures the drop in the maximized log-likelihood between the unrestricted and restricted models:

Computational Requirement: Requires fitting both the restricted and unrestricted models.

Theorem: Asymptotic Equivalence under the Null

Under standard Cramér regularity conditions, under the null hypothesis $H_0$:

All three test statistics share the identical limiting Chi-square distribution with $k$ degrees of freedom. Under local contiguous alternatives $\boldsymbol{\theta}_n = \boldsymbol{\theta}_0 + \mathbf{h} / \sqrt{n}$, all three tests achieve identical asymptotic power, forming the backbone of modern likelihood inference.

Can an estimator ever achieve an asymptotic variance strictly smaller than the Cramér–Rao bound $\frac{1}{I_1(\theta)}$? In 1951, Joseph Hodges constructed a famous counterexample proving that non-regular estimators can exhibit superefficiency at isolated parameter points.

Hodges' Superefficient Estimator

Let $X_1, \dots, X_n \stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(\theta, 1)$. Here $I_1(\theta) = 1$, so the classical CRLB is $1$. The sample mean $\bar{X}_n$ satisfies $\sqrt{n}(\bar{X}_n - \theta) \sim \mathcal{N}(0, 1)$ for all $\theta$. Define Hodges' estimator as:

Now analyze the limiting distribution of $\sqrt{n}(T_n - \theta)$:

1. Case 1 ($\theta \ne 0$):

Since $\bar{X}_n \xrightarrow{\text{a.s.}} \theta \ne 0$ and $n^{-1/4} \to 0$, the probability $\mathbb{P}(|\bar{X}_n| \le n^{-1/4}) \to 0$ exponentially fast. Thus $T_n = \bar{X}_n$ with probability approaching $1$, so:

The asymptotic variance is $V(\theta) = 1$, matching the Cramér–Rao bound.

2. Case 2 ($\theta = 0$):

Under $\theta = 0$, $\sqrt{n}\bar{X}_n \sim \mathcal{N}(0, 1)$, which means $\bar{X}_n = \mathcal{O}_P(n^{-1/2})$. Because $n^{-1/2} \ll n^{-1/4}$, the event $|\bar{X}_n| \le n^{-1/4}$ occurs with probability approaching $1$:

Thus $T_n = 0$ with probability approaching $1$! Consequently:

The limiting variance at $\theta = 0$ is:

At $\theta = 0$, Hodges' estimator has zero asymptotic variance—it is superefficient!

Le Cam's Resolution: Non-Uniform Convergence and Asymptotic Risk

Lucien Le Cam (1953) resolved this apparent paradox:

1. Measure Zero of Superefficiency: Le Cam proved that the set of parameter points where an estimator can be superefficient must have Lebesgue measure zero.

2. Catastrophic Penalty near the Superefficient Point: If one evaluates the risk not at a fixed point, but along a sequence of local drifting parameters $\theta_n = c / n^{1/4}$, the normalized mean squared error of Hodges' estimator diverges to infinity:

Superefficiency at $\theta = 0$ is purchased at the cost of an infinite risk spike in the immediate local neighborhood!

When sample sizes are moderate, analytical asymptotic Gaussian approximations $\mathcal{N}(0, I_n^{-1})$ can suffer from skewness and finite-sample distortions. Computer-intensive resampling methods provide non-parametric and parametric approximations to the sampling distribution of the MLE.

1. The Nonparametric Bootstrap (Efron, 1979)

Draw $B$ independent resamples $\mathbf{X}^{*b} = (X_1^{*b}, \dots, X_n^{*b})$ by sampling $n$ observations uniformly at random with replacement from the empirical dataset $\mathbf{x}$. For each bootstrap replicate, compute the MLE $\hat{\theta}^{*b}$. The bootstrap standard error is the empirical standard deviation of the bootstrap replicates:

2. The Jackknife (Quenouille–Tukey)

The delete-1 jackknife computes the parameter estimate $\hat{\theta}_{(-i)}$ on the subsample of size $n - 1$ obtained by omitting observation $X_i$. The jackknife bias estimate is:

and the jackknife variance estimator is:

where $\bar{\theta}_{(\cdot)} = \frac{1}{n} \sum_{i=1}^n \hat{\theta}_{(-i)}$. Jackknife and bootstrap estimators provide robust, second-order accurate variance estimates that do not require evaluating analytical score derivatives or Hessian matrices.

Superefficiency and the Le Cam–Hájek Asymptotic Minimax Theorem

In 1952, J. L. Hodges constructed a famous counterexample that challenged the classical belief that no consistent estimator could possess an asymptotic variance strictly smaller than the Cramér–Rao lower bound. Hodges defined the estimator:

where $X_i \sim \mathcal{N}(\theta, 1)$ and $|a| < 1$. When $\theta = 0$, $\sqrt{n}(T_n - 0) \to \mathcal{N}(0, a^2)$. Since $a^2 < 1$, the asymptotic variance at $\theta = 0$ is strictly less than $1$, the Cramér–Rao bound! Estimators exhibiting this property are termed superefficient.

This paradox was completely resolved by Lucien Le Cam and Jaroslav Hájek through the formulation of the Local Asymptotic Normality (LAN) framework and the Hájek–Le Cam Asymptotic Minimax Theorem. They demonstrated that:

1. The set of points where an estimator can achieve superefficiency must have Lebesgue measure zero in the parameter space $\Theta$.
2. Superefficiency at an isolated point $\theta_0$ is always purchased at the price of disastrously poor performance in a local shrinking neighborhood of the form $\theta_n = \theta_0 + h / \sqrt{n}$. In this neighborhood, the maximum risk of the superefficient estimator explodes relative to the regular maximum likelihood estimator.
3. The convolution theorem establishes that the asymptotic distribution of any regular estimator can be factored as the convolution of the optimal Gaussian limit with an independent noise distribution: $Z = \mathcal{N}(0, \mathbf{I}(\theta)^{-1}) + W$. Hence, the dispersion of the limit distribution is strictly minimized if and only if the noise term $W$ vanishes identically, which occurs uniquely for regular maximum likelihood estimators.