Foundations of Statistical Models, Parametric Families, Identifiability, and Likelihood Principles

At the heart of mathematical statistics lies the formal concept of a statistical experiment. In pure probability theory, one begins with a completely specified probability space $(\Omega, \mathcal{F}, P)$, where $\Omega$ represents the sample space of all conceivable outcomes, $\mathcal{F}$ is a $\sigma$-algebra of events (subsets of $\Omega$ satisfying closure under complementation and countable unions), and $P: \mathcal{F} \to [0, 1]$ is a known, fixed probability measure satisfying Kolmogorov's axioms: non-negativity, unit measure ($P(\Omega) = 1$), and countable additivity over pairwise disjoint collections of events. In that deductive framework, probability theory investigates the mathematical consequences of a known data-generating law, characterizing the behavior of random variables, deriving marginal and conditional distributions, and establishing asymptotic limit theorems.

Mathematical statistics fundamentally reverses this epistemological direction. In statistical inference, the physical or experimental reality generates an observed data vector $\mathbf{x} = (x_1, x_2, \dots, x_n) \in \mathcal{X}$, where $\mathcal{X}$ is the observation space (frequently a Borel measurable subset of Euclidean space $\mathbb{R}^n$). Crucially, the true probability measure $P$ governing the generation of $\mathbf{x}$ is unknown to the investigator. The primary objective of mathematical statistics is to make rigorous, mathematically justifiable assertions regarding the nature of the true, unobserved probability measure based exclusively on the realized observation $\mathbf{x}$.

Formally, a statistical model (or statistical experiment) is defined as a triplet:

where:

1. $\mathcal{X}$ denotes the sample space of possible data realizations.
2. $\mathcal{A}$ denotes a designated $\sigma$-algebra of observable events defined on $\mathcal{X}$, typically chosen as the Borel $\sigma$-algebra $\mathcal{B}(\mathcal{X})$ generated by the open topologies of Euclidean space.
3. $\mathcal{P} = \{P_\theta : \theta \in \Theta\}$ represents a specified family or collection of candidate probability measures defined on the measurable space $(\mathcal{X}, \mathcal{A})$, indexed by an arbitrary index set $\Theta$.

The statistical model $\mathcal{E}$ represents an explicit mathematical hypothesis formulated by the analyst. It asserts that the real-world random mechanism generating the data corresponds to some specific member $P_{\theta^*} \in \mathcal{P}$, where $\theta^* \in \Theta$ is the unknown "true" state of nature. The foundational challenge of estimation, hypothesis testing, and decision theory consists in identifying, approximating, or delimiting this true index $\theta^*$ from the empirical evidence provided by the realization $\mathbf{x} \in \mathcal{X}$.

If the family $\mathcal{P}$ cannot be indexed by a finite-dimensional Euclidean vector, the model is classified as nonparametric. For instance, if $\mathcal{P}$ is the collection of all continuous cumulative distribution functions on the real line $\mathbb{R}$, the parameter space $\Theta$ is an infinite-dimensional function space. If $\mathcal{P}$ involves both finite-dimensional parameters of structural interest (such as a treatment effect) and infinite-dimensional nuisance parameters (such as an arbitrary baseline hazard function in survival analysis), the model is designated as semiparametric. In this foundational volume of mathematical statistics, we concentrate principally on parametric statistical models, wherein the candidate measures are indexed by a finite-dimensional parameter vector $\boldsymbol{\theta} \in \Theta \subseteq \mathbb{R}^k$.

A parametric family of probability distributions arises when each candidate probability measure $P_{\boldsymbol{\theta}} \in \mathcal{P}$ is uniquely indexed by a parameter vector $\boldsymbol{\theta} = (\theta_1, \theta_2, \dots, \theta_k)^T$ residing within a specified parameter space $\Theta \subseteq \mathbb{R}^k$. The parameter space $\Theta$ is typically assumed to possess convenient topological structure; it is frequently an open, connected subset of Euclidean space $\mathbb{R}^k$, or the closure of such a set.

To facilitate mathematical analysis and analytical derivations, modern statistical theory almost universally assumes that the family $\mathcal{P} = \{P_{\boldsymbol{\theta}} : \boldsymbol{\theta} \in \Theta\}$ is dominated by a common $\sigma$-finite reference measure $\mu$ defined on $(\mathcal{X}, \mathcal{A})$. Under this dominance assumption, the Radon–Nikodym Theorem guarantees the existence of a non-negative, measurable function $f(\mathbf{x}; \boldsymbol{\theta})$ for every $\boldsymbol{\theta} \in \Theta$, such that for every observable event $A \in \mathcal{A}$:

The Radon–Nikodym derivative $f(\mathbf{x}; \boldsymbol{\theta}) = \frac{dP_{\boldsymbol{\theta}}}{d\mu}(\mathbf{x})$ represents the generalized probability density function (PDF) or probability mass function (PMF) of the random variable:

• If $\mathcal{X}$ is continuous ($\mathcal{X} \subseteq \mathbb{R}^n$) and $\mu$ is the standard Lebesgue measure $\lambda^n$, then $f(\mathbf{x}; \boldsymbol{\theta})$ is the classical probability density function, and integration is understood in the standard Riemann or Lebesgue sense:

• If $\mathcal{X}$ is discrete or countable and $\mu$ is the counting measure on $\mathcal{X}$, then $f(\mathbf{x}; \boldsymbol{\theta})$ represents the discrete probability mass function $f(\mathbf{x}; \boldsymbol{\theta}) = P_{\boldsymbol{\theta}}(\mathbf{X} = \mathbf{x})$, and integration reduces to countable summation:

The support of the distribution under parameter $\boldsymbol{\theta}$ is defined as the set of points where the density is strictly positive:

A crucial distinction in mathematical statistics exists between families with fixed support and families with parameter-dependent support:

1. Regular (Fixed Support) Families: The support $\mathcal{S}_{\boldsymbol{\theta}}$ is strictly invariant with respect to $\boldsymbol{\theta}$, so that $\mathcal{S}_{\boldsymbol{\theta}} = \mathcal{S}$ for all $\boldsymbol{\theta} \in \Theta$. Examples include the Gaussian distribution $\mathcal{N}(\mu, \sigma^2)$ on $\mathbb{R}$, the Exponential distribution $\text{Exp}(\lambda)$ on $(0, \infty)$, the Gamma distribution on $(0, \infty)$, and the Poisson distribution on $\{0, 1, 2, \dots\}$. In these families, differentiation with respect to $\boldsymbol{\theta}$ under the integral sign is generally permissible under standard dominated convergence conditions.

2. Non-Regular (Parameter-Dependent Support) Families: The boundary of the support $\mathcal{S}_{\boldsymbol{\theta}}$ explicitly depends on the parameter $\boldsymbol{\theta}$. The canonical example is the Uniform distribution $\text{Uniform}(0, \theta)$ on $\mathcal{S}_\theta = (0, \theta)$, or the shifted Exponential distribution on $(\theta, \infty)$. In non-regular families, standard likelihood regularity conditions collapse; differentiation under the integral sign across support boundaries produces Dirac delta distributions, requiring distinct non-standard asymptotic and inferential treatments (such as order-statistic maximum likelihood estimators with accelerated $\mathcal{O}(n)$ convergence rates rather than classical $\mathcal{O}(n^{1/2})$ rates).

Before any statistical procedure can be meaningfully constructed to estimate $\boldsymbol{\theta}$ or test hypotheses regarding its value, the parametric family must satisfy a fundamental structural property known as identifiability. Identifiability addresses whether distinct parameter values generate genuinely distinct probability distributions. If two different parameter values $\boldsymbol{\theta}_1 \ne \boldsymbol{\theta}_2$ yield mathematically identical probability distributions over the entire sample space, then no amount of empirical data—even an infinite sample size ($n \to \infty$)—can ever enable an investigator to distinguish $\boldsymbol{\theta}_1$ from $\boldsymbol{\theta}_2$.

Formal Definition of Identifiability

Let $\mathcal{P} = \{P_{\boldsymbol{\theta}} : \boldsymbol{\theta} \in \Theta\}$ be a family of probability measures defined on $(\mathcal{X}, \mathcal{A})$. The parameter $\boldsymbol{\theta}$ is said to be identifiable with respect to $\mathcal{P}$ if the mapping $\boldsymbol{\theta} \mapsto P_{\boldsymbol{\theta}}$ is injective (one-to-one). That is, for all $\boldsymbol{\theta}_1, \boldsymbol{\theta}_2 \in \Theta$:

Equivalently, taking the contrapositive, the parameter is identifiable if:

where $P_{\boldsymbol{\theta}_1} \ne P_{\boldsymbol{\theta}_2}$ means there exists at least one measurable event $A \in \mathcal{A}$ such that $P_{\boldsymbol{\theta}_1}(A) \ne P_{\boldsymbol{\theta}_2}(A)$. In terms of generalized probability density functions dominated by measure $\mu$, identifiability requires that if $\boldsymbol{\theta}_1 \ne \boldsymbol{\theta}_2$, the set $\{\mathbf{x} \in \mathcal{X} : f(\mathbf{x}; \boldsymbol{\theta}_1) \ne f(\mathbf{x}; \boldsymbol{\theta}_2)\}$ has strictly positive $\mu$-measure.

Mathematical Proof Techniques for Identifiability

To prove that a parametric family is identifiable, mathematicians deploy several rigorous analytical strategies depending on the analytic properties of the density:

1. Cumulative Distribution Function (CDF) Inversion: If $F(x; \boldsymbol{\theta}_1) = F(x; \boldsymbol{\theta}_2)$ for all $x \in \mathbb{R}$, evaluate limits or specific functional equations to deduce $\boldsymbol{\theta}_1 = \boldsymbol{\theta}_2$. For instance, in location-scale families where $F(x; \mu, \sigma) = F_0\left(\frac{x - \mu}{\sigma}\right)$, setting quantiles equal forces equality of location and scale parameters.

2. Transform Uniqueness Theorems (MGF and Characteristic Functions): If the moment generating function $M_X(t; \boldsymbol{\theta}) = \mathbb{E}_{\boldsymbol{\theta}}[e^{tX}]$ or characteristic function $\phi_X(t; \boldsymbol{\theta}) = \mathbb{E}_{\boldsymbol{\theta}}[e^{itX}]$ exists and is identical for all $t$ in a neighborhood of zero:

By the uniqueness theorem of Fourier–Stieltjes transforms, identical characteristic functions imply identical probability measures, allowing one to match coefficient expansions.

3. Moment Matching: If the distributions possess finite moments of all orders and are uniquely determined by their moments (satisfying the Carleman condition $\sum_{k=1}^\infty m_{2k}^{-1/(2k)} = \infty$), equating theoretical moments $\mathbb{E}_{\boldsymbol{\theta}_1}[X^k] = \mathbb{E}_{\boldsymbol{\theta}_2}[X^k]$ for $k = 1, 2, \dots$ often yields an algebraic system whose unique solution is $\boldsymbol{\theta}_1 = \boldsymbol{\theta}_2$.

Classical Counterexamples and Non-Identifiability Modes

To grasp the delicacy of identifiability, consider several canonical scenarios where identifiability fails:

1. Underdetermined Linear Regression (Collinearity / Rank Deficiency)

Consider the linear regression model $Y = \beta_1 X_1 + \beta_2 X_2 + \varepsilon$, where $\varepsilon \sim \mathcal{N}(0, \sigma^2)$. Suppose that in the population, the two explanatory variables are strictly collinear: $X_2 = 2 X_1$. The expected value of $Y$ given $X_1$ is:

The distribution of $Y$ given $(X_1, X_2)$ depends entirely on the scalar quantity $\gamma = \beta_1 + 2\beta_2$. Any pair $(\beta_1^*, \beta_2^*)$ satisfying $\beta_1^* + 2\beta_2^* = \gamma$ (such as $(c, (\gamma - c)/2)$ for any $c \in \mathbb{R}$) produces the exact same Gaussian conditional density $\mathcal{N}(\gamma x_1, \sigma^2)$. Hence, the individual parameters $(\beta_1, \beta_2)$ are completely non-identifiable. Only the linear combination $\gamma = \beta_1 + 2\beta_2$ is an identifiable parameter (known in linear model theory as an estimable function).

2. Latent Factor Rotation (Orthogonal Indeterminacy)

In factor analysis, an observed $p$-dimensional random vector $\mathbf{X}$ is modeled as $\mathbf{X} = \boldsymbol{\mu} + \mathbf{L} \mathbf{F} + \boldsymbol{\varepsilon}$, where $\mathbf{F} \sim \mathcal{N}_k(\mathbf{0}, \mathbf{I}_k)$ represents unobserved latent factors and $\mathbf{L} \in \mathbb{R}^{p \times k}$ is the factor loading matrix. The covariance matrix of $\mathbf{X}$ is:

where $\boldsymbol{\Psi} = \text{diag}(\sigma_1^2, \dots, \sigma_p^2)$. If $\mathbf{Q} \in \mathbb{R}^{k \times k}$ is any arbitrary orthogonal matrix (so that $\mathbf{Q} \mathbf{Q}^T = \mathbf{I}_k$), we can define rotated loadings $\mathbf{L}^* = \mathbf{L} \mathbf{Q}$. Then:

The covariance matrix $\boldsymbol{\Sigma}$, and therefore the entire multivariate Gaussian likelihood of $\mathbf{X}$, is completely invariant under the transformation $\mathbf{L} \mapsto \mathbf{L} \mathbf{Q}$. Thus, factor loadings are non-identifiable up to arbitrary orthogonal rotations without imposing identifying constraints (such as setting the upper triangle of $\mathbf{L}$ to zero or requiring $\mathbf{L}^T \boldsymbol{\Psi}^{-1} \mathbf{L}$ to be diagonal).

A profoundly important class of models where identifiability requires subtle mathematical care is the family of finite mixture distributions. Finite mixtures model populations composed of distinct, unobserved subpopulations (clusters).

Let $f_0(x; \boldsymbol{\phi})$ be a parametric family of densities. A $K$-component finite mixture density is expressed as:

where $\boldsymbol{\theta} = (\pi_1, \dots, \pi_K, \boldsymbol{\phi}_1, \dots, \boldsymbol{\phi}_K)$, the mixing proportions satisfy $\pi_k > 0$ and $\sum_{k=1}^K \pi_k = 1$, and $\boldsymbol{\phi}_k$ are distinct component parameters.

The Label Switching Problem

Finite mixture distributions are inherently non-identifiable under unconstrained permutation of indices. Let $\sigma: \{1, \dots, K\} \to \{1, \dots, K\}$ be any permutation of the integers $1, \dots, K$. Define the permuted parameter vector $\boldsymbol{\theta}_\sigma = (\pi_{\sigma(1)}, \dots, \pi_{\sigma(K)}, \boldsymbol{\phi}_{\sigma(1)}, \dots, \boldsymbol{\phi}_{\sigma(K)})$. Because addition of real functions is commutative:

For any parameter vector $\boldsymbol{\theta}$ with distinct components, there exist exactly $K!$ distinct parameter vectors in $\Theta$ that generate the identical probability density function! This phenomenon is known as the label switching problem.

In mathematical statistics, identifiability of finite mixtures is defined modulo permutations. We say a mixture family is identifiable if:

Teicher (1961, 1963) established celebrated theorems proving that finite mixtures of univariate Gaussian distributions and Gamma distributions are identifiable up to label switching. However, mixtures of Uniform distributions are generally non-identifiable.

Furthermore, severe boundary degeneracies occur in finite mixtures:

1. Zero-Weight Indeterminacy: If any component weight $\pi_k = 0$, the corresponding component parameter $\boldsymbol{\phi}_k$ can take any arbitrary value in its domain without changing the mixture density:

2. Component Merging Indeterminacy: If two component parameter vectors coincide ($\boldsymbol{\phi}_1 = \boldsymbol{\phi}_2 = \boldsymbol{\phi}^*$), the density becomes:

Any pair of weights satisfying $\pi_1 + \pi_2 = c$ produces the same distribution. Thus, along the manifold where component parameters merge, the individual weights $\pi_1$ and $\pi_2$ lose identifiability.

To resolve these ambiguities in applied mathematical statistics, analysts impose strict parameter ordering constraints, such as enforcing $\mu_1 < \mu_2 < \dots < \mu_K$ in Gaussian location mixtures, thereby restricting the parameter space to an identifiable fundamental domain.

Given a statistical model $\mathcal{E} = (\mathcal{X}, \mathcal{A}, \{P_{\boldsymbol{\theta}} : \boldsymbol{\theta} \in \Theta\})$ with generalized densities $f(\mathbf{x}; \boldsymbol{\theta})$ dominated by measure $\mu$, the mathematical role of the variables shifts completely once the experiment is executed.

Prior to the experiment, $\boldsymbol{\theta}$ is fixed and $\mathbf{x}$ is a random variable; $f(\mathbf{x}; \boldsymbol{\theta})$ is viewed as a function of $\mathbf{x} \in \mathcal{X}$, integrating to $1$:

Once the experiment is performed and a specific data realization $\mathbf{x} \in \mathcal{X}$ has been observed, $\mathbf{x}$ is fixed and invariant. We now view the function as mapping the parameter space $\Theta$ into non-negative real numbers.

Definition of the Likelihood Function

The Likelihood Function $L(\boldsymbol{\theta}; \mathbf{x}): \Theta \to [0, \infty)$ is defined as:

where $c(\mathbf{x}) > 0$ is an arbitrary positive constant that may depend on the observed data $\mathbf{x}$ but is strictly independent of the parameter $\boldsymbol{\theta}$.

Because the likelihood function is uniquely defined only up to an arbitrary multiplicative constant, the absolute numerical value $L(\boldsymbol{\theta}; \mathbf{x})$ is physically meaningless. The only mathematically meaningful quantities are likelihood ratios between competing candidate parameter values:

The likelihood ratio $\Lambda(\boldsymbol{\theta}_1, \boldsymbol{\theta}_2; \mathbf{x})$ quantifies the relative empirical plausibility of parameter $\boldsymbol{\theta}_1$ compared to $\boldsymbol{\theta}_2$ in light of the observed data $\mathbf{x}$. If $\Lambda(\boldsymbol{\theta}_1, \boldsymbol{\theta}_2; \mathbf{x}) = 10$, the observed data are ten times more probable under the hypothesis $\boldsymbol{\theta} = \boldsymbol{\theta}_1$ than under $\boldsymbol{\theta} = \boldsymbol{\theta}_2$.

The Log-Likelihood Function

In independent and identically distributed (i.i.d.) random sampling, where $\mathbf{x} = (x_1, \dots, x_n)$ and $f(\mathbf{x}; \boldsymbol{\theta}) = \prod_{i=1}^n f(x_i; \boldsymbol{\theta})$, the likelihood function is a product of $n$ terms. To convert computationally unstable products into analytically tractable sums, statisticians take the natural logarithm, defining the log-likelihood function:

Because the logarithm is a strictly monotonic increasing function on $(0, \infty)$, any local or global maximizer of $\ell(\boldsymbol{\theta}; \mathbf{x})$ over $\Theta$ is identically a local or global maximizer of $L(\boldsymbol{\theta}; \mathbf{x})$. Working with the log-likelihood brings decisive analytical advantages:

• Products decompose into summations, facilitating differentiation via linearity.
• The Weak and Strong Laws of Large Numbers and Central Limit Theorems apply directly to $\ell(\boldsymbol{\theta}; \mathbf{X}) = \sum_{i=1}^n \ln f(X_i; \boldsymbol{\theta})$, since it is a sum of i.i.d. random variables.

To locate the maximum likelihood estimator and examine the local geometry of the likelihood surface, we analyze its gradient with respect to $\boldsymbol{\theta}$.

Assume that $\Theta \subseteq \mathbb{R}^k$ is an open set and that the generalized density $f(\mathbf{x}; \boldsymbol{\theta})$ is continuously differentiable with respect to $\boldsymbol{\theta}$ on $\Theta$ for $\mu$-almost all $\mathbf{x}$.

Definition of the Score Vector

The Score Function (or score vector) $S(\boldsymbol{\theta}; \mathbf{x}) \in \mathbb{R}^k$ is the gradient of the log-likelihood function with respect to the parameter vector $\boldsymbol{\theta}$:

The score vector represents the sensitivity of the log-density to marginal perturbations in the parameter coordinates. When evaluated at a random sample $\mathbf{X} \sim P_{\boldsymbol{\theta}}$, the score $S(\boldsymbol{\theta}; \mathbf{X})$ is a random vector.

Bartlett's First Identity (Zero Expectation of the Score)

Under standard regularity conditions permitting the interchange of differentiation and integration, the expected value of the score vector under the true parameter $\boldsymbol{\theta}$ is identically zero:

Formal Proof:

By definition, the total probability of the sample space is unity for all $\boldsymbol{\theta} \in \Theta$:

Differentiating both sides with respect to the parameter vector $\boldsymbol{\theta}$:

Assuming standard Cramér–Rao regularity condition (R1)—that the support $\mathcal{S}$ is independent of $\boldsymbol{\theta}$ and that $\|\nabla_{\boldsymbol{\theta}} f(\mathbf{x}; \boldsymbol{\theta})\|$ is dominated by a $\mu$-integrable function $g(\mathbf{x})$—the Leibniz integral rule permits interchanging differentiation and integration:

Now, multiply and divide the integrand by $f(\mathbf{x}; \boldsymbol{\theta})$ on the support $\mathcal{S}$:

Recognizing the term in parentheses as the score function $S(\boldsymbol{\theta}; \mathbf{x})$:

This fundamental result is known as Bartlett's First Identity.

Martingale Property of Likelihood Scores

In sequential experimentation and time-series analysis, Bartlett's first identity generalizes into a profound stochastic property: the partial score process forms a martingale.

Let $X_1, X_2, \dots, X_n$ be sequentially observed random variables with joint filtration $\mathcal{F}_n = \sigma(X_1, \dots, X_n)$. The total log-likelihood is:

The total score $S_n(\boldsymbol{\theta}) = \sum_{i=1}^n s_i(\boldsymbol{\theta})$, where $s_i(\boldsymbol{\theta}) = \nabla_{\boldsymbol{\theta}} \ln f(X_i \mid \mathcal{F}_{i-1}; \boldsymbol{\theta})$. Conditioning on the past history $\mathcal{F}_{i-1}$, the conditional expectation of each increment is:

Therefore, the score sequence $\{S_n(\boldsymbol{\theta}), \mathcal{F}_n\}_{n \ge 1}$ is a mean-zero martingale:

This martingale structure enables mathematicians to establish asymptotic normality of maximum likelihood estimators in non-i.i.d., autoregressive, and continuous-time stochastic processes using Martingale Central Limit Theorems.

Because the score vector has zero expectation, its covariance matrix coincides with its second moment matrix. This covariance matrix is the celebrated Fisher Information Matrix.

Definition of Expected Fisher Information

The Expected Fisher Information Matrix $I(\boldsymbol{\theta}) \in \mathbb{R}^{k \times k}$ is defined as the covariance matrix of the score vector under the true parameter $\boldsymbol{\theta}$:

For a scalar parameter $\theta \in \mathbb{R}$, this simplifies to the variance:

Bartlett's Second Identity (Hessian Representation)

If the log-density is twice continuously differentiable with respect to $\boldsymbol{\theta}$ and integration and double differentiation can be interchanged, the Fisher Information matrix can be computed alternatively as the negative expected value of the Hessian matrix of the log-likelihood:

where the Hessian matrix has components $\mathcal{H}_{ij}(\boldsymbol{\theta}; \mathbf{x}) = \frac{\partial^2 \ln f(\mathbf{x}; \boldsymbol{\theta})}{\partial \theta_i \partial \theta_j}$.

Formal Proof of Bartlett's Second Identity:

From Bartlett's first identity, we know that for any coordinate $\theta_i$:

Differentiating this identity with respect to coordinate $\theta_j$:

Interchanging differentiation and integration by dominated convergence:

Applying the product rule of calculus to the integrand:

Notice that $\frac{\partial f(\mathbf{x}; \boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial \ln f(\mathbf{x}; \boldsymbol{\theta})}{\partial \theta_j} f(\mathbf{x}; \boldsymbol{\theta})$. Substituting this into the second term:

In expectation notation:

Rearranging yields Bartlett's second identity:

Observed Fisher Information

While the expected Fisher information $I(\boldsymbol{\theta})$ integrates over all potential realizations in the sample space, the Observed Fisher Information Matrix $J(\boldsymbol{\theta}; \mathbf{x})$ evaluates the negative Hessian directly at the observed data vector $\mathbf{x}$:

In applied inference and numerical optimization (such as Newton–Raphson), $J(\hat{\boldsymbol{\theta}}; \mathbf{x})$ evaluated at the maximum likelihood estimate $\hat{\boldsymbol{\theta}}$ measures the actual local curvature of the log-likelihood peak. If the log-likelihood surface is steeply curved (large negative eigenvalues of the Hessian), the observed information is large, indicating that the parameter is sharply pinned down by the data. Conversely, a flat, shallow likelihood ridge produces small observed information, signaling high parameter uncertainty.

Additivity of Fisher Information for Independent Samples

A foundational theorem of mathematical statistics is that Fisher information is strictly additive over independent experiments.

Let $X_1, X_2, \dots, X_n$ be independent random variables with densities $f_i(x_i; \boldsymbol{\theta})$. The joint log-likelihood is:

The joint score is the sum of individual scores:

Taking the variance of $S_n(\boldsymbol{\theta})$, because $X_i$ and $X_j$ are independent for $i \ne j$, their scores $S_i(\boldsymbol{\theta})$ and $S_j(\boldsymbol{\theta})$ are mutually independent and uncorrelated:

For an i.i.d. sample of size $n$ from a common density $f(x; \boldsymbol{\theta})$ with unit information $I_1(\boldsymbol{\theta})$, the total sample Fisher information scales linearly with sample size:

This linear growth in Fisher information is the mathematical engine driving the asymptotic convergence rate of maximum likelihood estimators: the asymptotic variance of $\hat{\boldsymbol{\theta}}$ scales inversely with sample size, decaying as $\mathcal{O}(1/n)$, which yields the standard $\sqrt{n}$ convergence rate $\sqrt{n}(\hat{\boldsymbol{\theta}} - \boldsymbol{\theta}) \xrightarrow{d} \mathcal{N}(\mathbf{0}, I_1(\boldsymbol{\theta})^{-1})$.

The validity of Bartlett's identities, the Cramér–Rao Lower Bound, and the asymptotic normality of maximum likelihood estimators rests upon a collection of technical assumptions known as the Cramér Regularity Conditions.

The Classical Regularity Conditions (Cramér–Rao Assumptions):

1. Identifiability (C1): The parameter space $\Theta$ is an open subset of $\mathbb{R}^k$, and distinct parameter values index distinct probability measures: $\boldsymbol{\theta}_1 \ne \boldsymbol{\theta}_2 \implies P_{\boldsymbol{\theta}_1} \ne P_{\boldsymbol{\theta}_2}$.

2. Common Support (C2): The support $\mathcal{S} = \{\mathbf{x} \in \mathcal{X} : f(\mathbf{x}; \boldsymbol{\theta}) > 0\}$ is strictly independent of $\boldsymbol{\theta}$. This ensures that support boundaries do not depend on parameters.

3. Differentiability (C3): For $\mu$-almost all $\mathbf{x} \in \mathcal{X}$, the density $f(\mathbf{x}; \boldsymbol{\theta})$ is continuously differentiable with respect to $\boldsymbol{\theta}$ up to the third order on an open neighborhood $\mathcal{U} \subset \Theta$.

4. Interchange of Differentiation and Integration (C4): For every $\boldsymbol{\theta} \in \Theta$, there exist $\mu$-integrable bounding functions permitting the exchange of derivative and integral signs:

5. Positive-Definite Information (C5): For all $\boldsymbol{\theta} \in \Theta$, the Fisher Information matrix $I(\boldsymbol{\theta})$ is strictly positive-definite:

This guarantees that $I(\boldsymbol{\theta})$ is invertible and non-singular.

6. Third-Derivative Domination (C6): There exists a measurable function $M(\mathbf{x})$ and a neighborhood $\mathcal{U}$ of the true parameter $\boldsymbol{\theta}_0$ such that:

where $\mathbb{E}_{\boldsymbol{\theta}_0}[M(\mathbf{X})] < \infty$. This ensures that the remainder term in a Taylor series expansion of the score function is asymptotically negligible.

Information Geometry and the Rao Distance

In 1945, C. R. Rao made the profound mathematical discovery that the Fisher Information matrix defines a Riemannian metric tensor on the manifold of probability distributions $\mathcal{M} = \{P_{\boldsymbol{\theta}} : \boldsymbol{\theta} \in \Theta\}$.

In differential geometry, a manifold is equipped with an inner product on its tangent space at each point $\boldsymbol{\theta}$. For two infinitesimal tangent vectors $d\boldsymbol{\theta}_1, d\boldsymbol{\theta}_2$, the Fisher Information matrix acts as the Riemannian metric:

The infinitesimal square distance between two neighboring probability distributions $P_{\boldsymbol{\theta}}$ and $P_{\boldsymbol{\theta} + d\boldsymbol{\theta}}$ is given by the differential quadratic form:

Remarkably, this metric distance is directly connected to information theory: it is equal to twice the infinitesimal Kullback–Leibler divergence between the two distributions:

The shortest path between two probability distributions on the manifold $\mathcal{M}$ is a geodesic curve $\boldsymbol{\theta}(t)$, and the length of this geodesic is the Rao Distance:

The Rao distance provides an intrinsic, geometrically invariant measure of the statistical distinguishability between two probability models, independent of the particular coordinate system chosen to parameterize the family.

In realistic statistical applications, the parameter vector naturally partitions into two subsets:

where:

• $\boldsymbol{\psi} \in \Psi \subseteq \mathbb{R}^p$ represents the parameter of interest (such as a treatment effect, physical constant, or causal hazard ratio).
• $\boldsymbol{\lambda} \in \Lambda \subseteq \mathbb{R}^{k - p}$ represents a vector of nuisance parameters (such as measurement error variance, baseline hazard, or spatial correlation parameters) that are not of primary scientific interest but must be incorporated to ensure valid modeling of the physical reality.

When nuisance parameters are present, standard inferential methods require eliminating $\boldsymbol{\lambda}$ while preserving accurate uncertainty quantification for $\boldsymbol{\psi}$.

The Profile Likelihood

The most widely employed classical tool for nuisance parameter elimination is the Profile Likelihood Function.

Definition:

For each fixed value of the parameter of interest $\boldsymbol{\psi}$, the profile likelihood $L_p(\boldsymbol{\psi}; \mathbf{x})$ is obtained by maximizing the full likelihood function over all possible values of the nuisance parameter vector $\boldsymbol{\lambda}$:

where $\hat{\boldsymbol{\lambda}}_{\boldsymbol{\psi}}$ denotes the constrained maximum likelihood estimator of $\boldsymbol{\lambda}$ given a fixed value of $\boldsymbol{\psi}$:

The corresponding profile log-likelihood is:

Profile Score and Profile Information

By the multivariable chain rule, the gradient of the profile log-likelihood with respect to $\boldsymbol{\psi}$ is:

By definition of the constrained maximum likelihood estimator, $\hat{\boldsymbol{\lambda}}_{\boldsymbol{\psi}}$ satisfies the first-order stationarity condition $\left. \nabla_{\boldsymbol{\lambda}} \ell(\boldsymbol{\psi}, \boldsymbol{\lambda}) \right|_{\boldsymbol{\lambda} = \hat{\boldsymbol{\lambda}}_{\boldsymbol{\psi}}} = \mathbf{0}$. Therefore, the second term vanishes identically!

The profile score is simply the ordinary score vector evaluated at the constrained nuisance estimate.

Partitioning the full observed information matrix $J(\boldsymbol{\theta})$ conformably with $\boldsymbol{\psi}$ and $\boldsymbol{\lambda}$:

The negative second derivative of the profile log-likelihood yields the Profile Observed Information:

Remarkably, $J_{\psi\psi} - J_{\psi\lambda} J_{\lambda\lambda}^{-1} J_{\lambda\psi}$ is exactly the Schur complement of $J_{\lambda\lambda}$ in the block matrix $J(\boldsymbol{\theta})$. By block matrix inversion identities, the inverse of the Schur complement is equal to the upper-left block of the inverted full information matrix:

This proves that profile likelihood curvature automatically incorporates the inflation in estimation variance caused by not knowing the true nuisance parameter vector $\boldsymbol{\lambda}$!

Limitations of Profile Likelihood: The Neyman–Scott Paradox

While profile likelihood is asymptotically valid when the dimension of the nuisance parameter $q = \dim(\boldsymbol{\lambda})$ remains fixed as $n \to \infty$, it can fail catastrophically when the number of nuisance parameters grows proportionally with sample size ($q \propto n$).

This failure is immortalized by the Neyman–Scott Paradox (1948). Suppose we observe $n$ independent pairs:

Here, $\sigma^2$ is the parameter of interest ($p = 1$), while the pair-specific means $\boldsymbol{\mu} = (\mu_1, \dots, \mu_n)$ are nuisance parameters ($q = n$).

The log-likelihood function is:

For any fixed $\sigma^2$, maximizing over $\mu_i$ yields the constrained estimator $\hat{\mu}_i = \bar{X}_i = \frac{X_{i1} + X_{i2}}{2}$. Substituting $\hat{\mu}_i$ into the log-likelihood produces the profile log-likelihood for $\sigma^2$:

Differentiating $\ell_p(\sigma^2)$ and setting to zero yields the profile maximum likelihood estimator:

Since $X_{i1} - X_{i2} \sim \mathcal{N}(0, 2\sigma^2)$, the expected value is:

By the Strong Law of Large Numbers, as $n \to \infty$:

The profile maximum likelihood estimator converges to half the true variance! It is strictly inconsistent, underestimating variance by $50\%$. The failure occurs because the profile likelihood fails to adjust for the degrees of freedom consumed by estimating $n$ separate nuisance parameters. In modern statistics, this limitation is overcome using Marginal Likelihood, Conditional Likelihood, or Restricted Maximum Likelihood (REML).

The Likelihood Principle (LP) represents one of the most intellectually fascinating and contentious philosophical cornerstones of foundational statistics. Formulated informally by R. A. Fisher and George Barnard, and given axiomatic derivation by Allan Birnbaum (1962), the Likelihood Principle asserts:

The Likelihood Principle: All the evidence that an experimental realization $\mathbf{x}$ provides regarding the unknown parameter $\boldsymbol{\theta}$ is contained entirely in the likelihood function $L(\boldsymbol{\theta}; \mathbf{x})$. If two distinct experiments involving the same parameter $\boldsymbol{\theta}$ yield proportional likelihood functions for realized observations $\mathbf{x}_1$ and $\mathbf{x}_2$:

then any statistical inference regarding $\boldsymbol{\theta}$ drawn from $\mathbf{x}_1$ must be identically equal to the inference drawn from $\mathbf{x}_2$.

Birnbaum's Equivalence Theorem

Birnbaum proved that the Likelihood Principle is mathematically equivalent to the confluence of two widely accepted, seemingly innocuous principles of statistical inference:

1. The Sufficiency Principle (SP): If $T(\mathbf{X})$ is a sufficient statistic for $\boldsymbol{\theta}$, and $T(\mathbf{x}_1) = T(\mathbf{x}_2)$, then the experimental evidence regarding $\boldsymbol{\theta}$ provided by $\mathbf{x}_1$ and $\mathbf{x}_2$ is identical.

2. The Conditionality Principle (CP): If an experiment is selected at random from a collection of candidate experiments by a mechanism whose probability distribution is strictly independent of $\boldsymbol{\theta}$ (such as flipping a fair coin to decide whether to run Experiment 1 or Experiment 2), then once the coin has landed and Experiment $j$ is performed yielding $\mathbf{x}_j$, all valid statistical inferences about $\boldsymbol{\theta}$ must depend solely on the performed Experiment $j$ and its outcome $\mathbf{x}_j$, ignoring the unperformed experiment.

Birnbaum demonstrated the theorem:

The Stopping Rule Paradox

The radical implication of the Likelihood Principle is that the experimental stopping rule must be irrelevant to inference about $\boldsymbol{\theta}$.

Consider two researchers investigating the success probability $\theta \in (0, 1)$ of a new medical treatment:

• Experiment 1 (Binomial Sampling): Researcher A decides in advance to treat exactly $n = 12$ patients. She observes $k = 3$ successes and $9$ failures.

The distribution of successes $K \sim \text{Binomial}(12, \theta)$:

• Experiment 2 (Negative Binomial / Inverse Sampling): Researcher B decides in advance to continue treating patients until exactly $r = 3$ successes are observed, then stop. He happens to stop after treating the $12\text{th}$ patient, observing $3$ successes and $9$ failures.

The total number of trials $N \sim \text{NegativeBinomial}(3, \theta)$:

Notice that the likelihood functions for both experiments are identical up to a scalar constant:

According to the Likelihood Principle, because the likelihood functions are proportional, both researchers must draw identical inferences about $\theta$.

However, in standard frequentist inference, the $p$-value for testing the null hypothesis $H_0: \theta \le 0.1$ against $H_1: \theta > 0.1$ depends on the probability of observing data "at least as extreme as" the observed outcome in hypothetical repetitions under the chosen stopping rule:

• For Researcher A (Binomial):

Since $p_A > 0.05$, Researcher A fails to reject $H_0$.

• For Researcher B (Negative Binomial):

Observing an outcome "at least as extreme" in favor of large $\theta$ means requiring $12$ or fewer trials to achieve $3$ successes ($N \le 12$):

Since $p_B < 0.05$, Researcher B rejects $H_0$ at the $5\%$ significance level!

This disparity reveals a fundamental philosophical divide in mathematical statistics:

• Frequentist inference (including confidence intervals, $p$-values, and significance levels) violates the strong Likelihood Principle, because frequentist error probabilities integrate over hypothetical unobserved outcomes in the sample space $\mathcal{X}$, which inherently depend on the sampling plan and stopping intention.
• Bayesian inference and pure Likelihood inference strictly adhere to the Likelihood Principle: since posterior distributions and likelihood ratios depend only on the observed data likelihood $L(\theta; \mathbf{x})$, both experiments produce identical posterior distributions and identical point estimates.

In statistical theory, parameters are mere mathematical coordinate systems used to label probability distributions. A physical theory should not depend on whether a temperature distribution is parameterized in Celsius or Kelvin, or whether a Gaussian distribution is parameterized by variance $\sigma^2$, standard deviation $\sigma$, or precision $\tau = 1/\sigma^2$.

Let $\boldsymbol{\theta} \in \Theta \subseteq \mathbb{R}^k$ be an identifiable parameter vector, and let $\mathbf{g}: \Theta \to \Omega \subseteq \mathbb{R}^k$ be a smooth, invertible bijection with differentiable inverse $\mathbf{h} = \mathbf{g}^{-1}: \Omega \to \Theta$.

Let $\boldsymbol{\eta} = \mathbf{g}(\boldsymbol{\theta})$ denote the new parameter vector, so that $\boldsymbol{\theta} = \mathbf{h}(\boldsymbol{\eta})$. The reparameterized probability density function is:

The Jacobian Transformation Matrix

Let $\mathbf{J}_{\mathbf{h}}(\boldsymbol{\eta}) \in \mathbb{R}^{k \times k}$ denote the Jacobian matrix of the inverse transformation $\mathbf{h}$:

By the multivariable chain rule, the score vector in the new coordinate system $\boldsymbol{\eta}$ relates to the original score vector via:

In compact vector notation:

The score vector transforms as a covariant vector (or differential one-form) under coordinate changes.

Transformation of the Fisher Information Matrix

Applying the definition of the Fisher Information matrix as the covariance of the score vector:

By properties of the covariance operator under linear matrix transformations:

This establishes the tensor transformation law for Fisher Information:

In component notation:

This transformation rule confirms that the Fisher Information matrix behaves strictly as a second-order covariant tensor metric, exactly fulfilling the axioms of Riemannian geometry.