Cramér–Rao Inequality, Fisher Information & Efficiency

To establish mathematical bounds on estimation precision, we must ensure that the likelihood function is sufficiently well-behaved to permit the interchange of differentiation and integration across the sample space. Let $\mathcal{P} = \{f(\mathbf{x}; \theta) : \theta \in \Theta\}$ be a family of probability density functions dominated by a $\sigma$-finite reference measure $\mu$ on $(\mathcal{X}, \mathcal{A})$, where the parameter space $\Theta$ is an open interval or open connected domain in $\mathbb{R}$.

The classical Cramér–Rao Regularity Conditions (often denoted R0 through R4) are formulated as follows:

1. Open Parameter Domain (R0): The parameter space $\Theta \subseteq \mathbb{R}$ is an open interval. This ensures that every parameter point $\theta$ is an interior point, admitting two-sided neighborhoods for differentiation.

2. Invariant Support Condition (R1): The support of the density:

is strictly independent of the parameter $\theta$. If the support boundary depends on $\theta$ (as in the uniform distribution $\text{Uniform}(0, \theta)$ or shifted exponential distribution), Leibniz's rule across boundary lines introduces Dirac delta distributions, invalidating classical information bounds.

3. Differentiability of the Density (R2): For $\mu$-almost all $\mathbf{x} \in \mathcal{X}$, the derivative $\frac{\partial}{\partial \theta} f(\mathbf{x}; \theta)$ exists and is continuous for all $\theta \in \Theta$.

4. Interchange of Differentiation and Integration (R3): The total probability integral:

can be differentiated with respect to $\theta$ under the integral sign:

Similarly, for any estimator $T(\mathbf{X})$ with finite expectation $\mathbb{E}_\theta[T] = \int_\mathcal{X} T(\mathbf{x}) f(\mathbf{x}; \theta) \, d\mu(\mathbf{x})$, differentiation under the integral sign is permissible:

By the dominated convergence theorem, condition (R3) is guaranteed if there exist $\mu$-integrable functions $g_1(\mathbf{x})$ and $g_2(\mathbf{x})$ such that $\left| \frac{\partial f}{\partial \theta} \right| \le g_1(\mathbf{x})$ and $\left| T(\mathbf{x}) \frac{\partial f}{\partial \theta} \right| \le g_2(\mathbf{x})$ for all $\theta$ in a neighborhood of interest.

5. Positive and Finite Fisher Information (R4): The Fisher Information $I(\theta)$ satisfies:

A zero information value implies that data provide zero discriminatory evidence regarding perturbations in $\theta$, whereas infinite information signals non-regular super-efficiency.

Pathology of Support-Dependent Densities: The Uniform Model

To comprehend why condition (R1) is indispensable, consider an i.i.d. sample $X_1, \dots, X_n \sim \text{Uniform}(0, \theta)$ with density $f(x; \theta) = \frac{1}{\theta} \mathbf{1}_{(0, \theta)}(x)$. The joint density is:

If one naively ignores the boundary and applies the score formula:

This would suggest a Cramér–Rao lower bound of $\frac{\theta^2}{n}$. However, the maximum order statistic $X_{(n)}$ has distribution function $F_{(n)}(t) = (t/\theta)^n$, and its expectation is $\mathbb{E}[X_{(n)}] = \frac{n}{n+1} \theta$. Therefore, the scaled statistic:

is strictly unbiased for $\theta$. Its variance is:

Notice that for $n \ge 2$:

The variance of $T^*$ is strictly smaller than the supposed Cramér–Rao bound! Indeed, as $n \to \infty$, the variance decays at rate $\mathcal{O}(1/n^2)$, far faster than the regular parametric rate $\mathcal{O}(1/n)$. The breakdown occurs because Leibniz's integral rule for variable boundaries states:

The boundary term $T(\theta) f(\theta; \theta) = \frac{T(\theta)}{\theta} \ne 0$ introduces a non-zero jump that completely destroys Bartlett's first identity and the Cauchy–Schwarz covariance derivation. Hence, standard information inequalities strictly require invariant supports.

The central mathematical object mediating information inequalities is the Score Function. For a sample $\mathbf{X} \in \mathcal{X}$, the score function $S(\theta; \mathbf{X})$ is defined as the gradient of the log-likelihood function:

When evaluated at the true parameter $\theta$, the score is a random variable reflecting the localized rate of change of the probability density.

Bartlett's First Identity: Zero Mean of the Score

Under regularity conditions (R0)-(R3), the expectation of the score function is identically zero across the entire parameter space:

Proof:

Differentiate both sides of the identity $\int_\mathcal{X} f(\mathbf{x}; \theta) \, d\mu(\mathbf{x}) = 1$ with respect to $\theta$:

Multiply and divide the integrand by $f(\mathbf{x}; \theta)$ on the support $\mathcal{S}$:

Fisher Information

Because the score has expectation zero, its variance equals its second raw moment. The Fisher Information $I(\theta)$ contained in the sample $\mathbf{X}$ is defined as:

Bartlett's Second Identity: Curvature Representation of Information

If the density is twice continuously differentiable with respect to $\theta$ and differentiation can be interchanged twice with integration, Fisher Information can be computed alternatively as the negative expectation of the second derivative of the log-likelihood (the curvature of the log-likelihood surface):

Proof:

Differentiate Bartlett's first identity $\int_\mathcal{X} \frac{\partial \ln f(\mathbf{x}; \theta)}{\partial \theta} f(\mathbf{x}; \theta) \, d\mu(\mathbf{x}) = 0$ with respect to $\theta$:

Interchanging derivative and integral:

Apply the product rule of calculus to the integrand:

Substitute $\frac{\partial f(\mathbf{x}; \theta)}{\partial \theta} = \frac{\partial \ln f(\mathbf{x}; \theta)}{\partial \theta} f(\mathbf{x}; \theta)$ into the second term:

Writing in expectation notation:

Rearranging immediately yields:

This curvature representation is profoundly useful both computationally and conceptually. When the log-likelihood peak is sharply pointed and highly curved, its second derivative is large and negative, meaning that the Fisher Information is large, data discriminate sharply between candidate parameters, and estimators can achieve high precision.

Additivity of Fisher Information for Independent Observations

If $\mathbf{X} = (X_1, X_2, \dots, X_n)$ is an i.i.d. random sample with common density $f_1(x; \theta)$, the joint log-likelihood is:

The joint score function is the sum of the individual scores:

Because the observations are mutually independent, the score components $S_1(\theta; X_i)$ are independent random variables with mean zero. Taking the variance:

The total sample Fisher Information grows strictly linearly with sample size $n$.

We now present the rigorous proof of the Cramér–Rao Lower Bound using the geometric structure of the Hilbert space $L^2(P_\theta)$ of square-integrable random variables equipped with the inner product $\langle U, V \rangle = \mathbb{E}_\theta[UV]$.

Theorem: The Cramér-Rao Lower Bound (Scalar Parameter)

Let $\mathbf{X} \sim f(\mathbf{x}; \theta)$ satisfy the regularity conditions (R0)-(R4). Let $T(\mathbf{X})$ be any estimator of a differentiable parametric function $\tau(\theta)$ such that $\mathbb{E}_\theta[T^2] < \infty$. Let $m(\theta) = \mathbb{E}_\theta[T(\mathbf{X})]$. Then:

In particular, if $T(\mathbf{X})$ is an unbiased estimator of $\tau(\theta)$, so that $m(\theta) = \tau(\theta)$ and $m'(\theta) = \tau'(\theta)$, then:

If the target parameter is $\theta$ itself, $\tau(\theta) = \theta$, $\tau'(\theta) = 1$, and the lower bound simplifies to:

Proof:

Consider the covariance between the estimator $T(\mathbf{X})$ and the score function $S_n(\theta; \mathbf{X})$:

Since $\mathbb{E}_\theta[S_n] = 0$ by Bartlett's first identity:

By regularity condition (R3), we can interchange differentiation and integration:

Now, by the Cauchy–Schwarz inequality for real random variables:

Substitute the derived covariance $\text{Cov}_\theta(T, S_n) = m'(\theta)$ and score variance $\text{Var}_\theta(S_n) = I_n(\theta)$:

Dividing both sides by $I_n(\theta) > 0$ yields the celebrated inequality:

This completes the proof.

Geometric Interpretation in Hilbert Space $L^2(P_\theta)$

The Cauchy–Schwarz derivation admits an illuminating geometric interpretation within the Hilbert space of zero-mean square-integrable random variables $\mathcal{H} = \{ U : \mathbb{E}_\theta[U] = 0, \mathbb{E}_\theta[U^2] < \infty \}$ equipped with inner product $\langle U, V \rangle = \text{Cov}_\theta(U, V) = \mathbb{E}_\theta[UV]$ and norm $\|U\| = \sqrt{\text{Var}_\theta(U)}$.

In this space:

• The centered estimator $T_0(\mathbf{X}) = T(\mathbf{X}) - m(\theta)$ is a vector in $\mathcal{H}$ with length $\|T_0\| = \sqrt{\text{Var}_\theta(T)}$.
• The sample score function $S_n(\theta; \mathbf{X})$ is a vector in $\mathcal{H}$ with length $\|S_n\| = \sqrt{I_n(\theta)}$.
• The inner product is $\langle T_0, S_n \rangle = \text{Cov}_\theta(T, S_n) = m'(\theta)$.

The correlation coefficient between the estimator and the score is the cosine of the angle $\phi$ between these two vectors in $\mathcal{H}$:

Because $|\cos(\phi)| \le 1$ for any geometric angle, squaring both sides yields:

The efficiency of an unbiased estimator is nothing other than $\text{eff}_\theta(T) = \cos^2(\phi)$! An estimator is fully efficient ($\text{eff}_\theta(T) = 1$) if and only if the angle $\phi = 0$ or $\pi$, which means the estimator vector lies directly along the one-dimensional subspace spanned by the score function vector. Any component of $T_0$ orthogonal to the score function represents extraneous, uninformative sampling variability that inflates the variance without contributing to the estimation of $\theta$.

The Cramér–Rao inequality extends naturally to biased estimators. Suppose an estimator $T(\mathbf{X})$ has bias $b(\theta) = \mathbb{E}_\theta[T] - \theta$. Then its expectation is:

Differentiating with respect to $\theta$:

Substituting into the general Cramér–Rao bound yields:

Consequently, the Mean Squared Error of any biased estimator satisfies the inequality:

The Derivative of the Bias and Super-Precision

Notice an essential mathematical subtlety: if an estimator has a negative bias slope $b'(\theta) < 0$, the numerator $[1 + b'(\theta)]^2$ is strictly less than $1$! For example, if $b'(\theta) = -0.5$, then $[1 + b'(\theta)]^2 = 0.25$, allowing the variance of $T$ to be four times smaller than $\frac{1}{I_n(\theta)}$.

Deep Comparison of Biased Sample Variance Estimators

To observe the biased Cramér–Rao bound in action, consider an i.i.d. normal sample $X_1, \dots, X_n \sim \mathcal{N}(\mu, \sigma^2)$ where $\mu$ is unknown. Consider the family of estimators for $\sigma^2$:

Because $\frac{\sum (X_i - \bar{X})^2}{\sigma^2} \sim \chi_{n-1}^2$, we know that $\mathbb{E}[S_k^2] = \frac{n-1}{k} \sigma^2$ and $\text{Var}(S_k^2) = \frac{2(n-1)}{k^2} \sigma^4$.

1. Unbiased Estimator ($k = n - 1$):

Notice that the single-sample Fisher Information for $\sigma^2$ is $I_1(\sigma^2) = \frac{1}{2\sigma^4}$, so the unbiased CRLB is $\frac{2\sigma^4}{n}$. The variance of $S_{n-1}^2$ is $\frac{2\sigma^4}{n-1} > \frac{2\sigma^4}{n}$ because $\mu$ is unknown (a 1 degree of freedom penalty for estimating $\mu$).

2. Maximum Likelihood Estimator ($k = n$):

The biased Cramér–Rao lower bound for an estimator with this expectation $m(\sigma^2) = \frac{n-1}{n} \sigma^2$ is:

The actual variance is $\text{Var}(S_n^2) = \frac{2(n-1)}{n^2} \sigma^4$. The Mean Squared Error of the MLE is:

3. Minimum MSE Shrinkage Estimator ($k = n + 1$):

Minimizing $\text{MSE}(S_k^2) = \frac{2(n-1)}{k^2} \sigma^4 + \left(\frac{n-1-k}{k}\right)^2 \sigma^4$ with respect to $k$ yields $k^* = n + 1$. The resulting estimator $S_{n+1}^2$ achieves the strictly minimal MSE:

The biased estimator $S_{n+1}^2$ completely dominates the classical unbiased estimator $S_{n-1}^2$ across every single sample size $n \ge 2$ and for every true variance $\sigma^2 > 0$!

When does an unbiased estimator achieve the Cramér–Rao Lower Bound with strict mathematical equality? Understanding this question connects information theory directly to the structure of exponential families.

Theorem: Exact CRLB Attainment Condition

Equality holds in the Cauchy–Schwarz inequality if and only if one of the random variables is a deterministic linear transformation of the other almost everywhere. Therefore, for an unbiased estimator $T(\mathbf{X})$ of $\tau(\theta)$, equality:

holds for all $\theta \in \Theta$ if and only if there exists a deterministic scalar function $k(\theta)$ (independent of $\mathbf{x}$) such that the score function can be written as:

Furthermore, integrating both sides of this equation with respect to $\theta$ reveals the exact analytical form of the underlying probability distribution:

Exponentiating both sides yields:

where $\eta(\theta) = \int k(\theta) \, d\theta$ and $B(\theta) = \int k(\theta) \tau(\theta) \, d\theta$.

Conclusion on Attainment

This is the canonical definition of a one-parameter natural exponential family! We have established a profound foundational theorem: An unbiased estimator $T(\mathbf{X})$ can achieve the Cramér–Rao Lower Bound if and only if the underlying distribution belongs to the one-parameter exponential family, and the parameter being estimated is $\tau(\theta) = \mathbb{E}_\theta[T(\mathbf{X})]$.

If the distribution does not belong to an exponential family (e.g., Cauchy or Student's $t$), or if the parametric target $\tau(\theta)$ is a non-linear function of the natural parameter expectation (e.g., estimating $\sigma = \sqrt{\sigma^2}$ in a Gaussian model), the CRLB is strictly unattainable for finite samples! In such cases, the UMVUE exists, but its finite-sample variance lies strictly above the CRLB.

In real-world scientific problems, experiments typically involve multiple unknown parameters simultaneously, such as the mean and variance $(\mu, \sigma^2)$ of a Gaussian distribution, or multiple regression coefficients $\boldsymbol{\beta} \in \mathbb{R}^p$.

Let $\boldsymbol{\theta} = (\theta_1, \theta_2, \dots, \theta_k)^T \in \Theta \subseteq \mathbb{R}^k$. The Score Vector is defined as the gradient:

By multivariate Bartlett identities, $\mathbb{E}_{\boldsymbol{\theta}}[\mathbf{S}] = \mathbf{0}$. The Fisher Information Matrix $\mathbf{I}(\boldsymbol{\theta}) \in \mathbb{R}^{k \times k}$ is the $k \times k$ covariance matrix of the score vector:

with matrix entries:

Theorem: The Multiparameter Cramér-Rao Inequality

Let $\mathbf{T}(\mathbf{X}) = (T_1, \dots, T_m)^T$ be an unbiased estimator of a vector parametric function $\boldsymbol{\tau}(\boldsymbol{\theta}) = (\tau_1(\boldsymbol{\theta}), \dots, \tau_m(\boldsymbol{\theta}))^T$. Let $\mathbf{D}(\boldsymbol{\theta}) \in \mathbb{R}^{m \times k}$ denote the Jacobian matrix of partial derivatives:

Assume that the Fisher Information matrix $\mathbf{I}(\boldsymbol{\theta})$ is non-singular and invertible. Then the covariance matrix of $\mathbf{T}(\mathbf{X})$, denoted $\boldsymbol{\Sigma}_{\mathbf{T}}$, satisfies the matrix Loewner order inequality:

That is, the difference matrix is positive semi-definite. In particular, for estimating a single scalar parameter $\theta_i$ in the presence of nuisance parameters $\theta_j$ ($j \ne i$), the lower bound is given by the diagonal element of the inverted Fisher Information matrix:

The Cost of Nuisance Parameters

A remarkable matrix property holds for any positive-definite symmetric block matrix. Partition the Fisher Information matrix as:

By the block matrix inversion theorem (Schur complement):

Because $\mathbf{I}_{22}$ is positive-definite, the quadratic form $\mathbf{I}_{12} \mathbf{I}_{22}^{-1} \mathbf{I}_{21} \ge 0$. Consequently:

Parameter Orthogonality and the Cox–Reid Transformation

In statistical design and inference, two parameters $\theta_1$ and $\theta_2$ are defined to be information-orthogonal if their off-diagonal Fisher Information entry vanishes identically:

When parameters are orthogonal, the Fisher Information matrix is block-diagonal. This orthogonality yields several profound inferential benefits:

1. Zero Efficiency Penalty: The Cramér–Rao Lower Bound for $\theta_1$ in the presence of unknown $\theta_2$ is identical to the bound when $\theta_2$ is known: $[\mathbf{I}(\boldsymbol{\theta})^{-1}]_{11} = \frac{1}{I_{11}}$.

2. Asymptotic Independence: Maximum likelihood estimators $\hat{\theta}_1$ and $\hat{\theta}_2$ are asymptotically independent and uncorrelated.

3. Robustness to Nuisance Mis-specification: Slight errors in estimating $\theta_2$ exert second-order rather than first-order effects on the estimation of $\theta_1$.

Cox and Reid (1987) demonstrated that for any scalar parameter of interest $\theta_1$ and scalar nuisance parameter $\lambda$, one can systematically construct an orthogonal reparameterization $\psi = \psi(\theta_1, \lambda)$ by solving the partial differential equation:

Integrating this characteristic equation yields an orthogonal coordinate system where nuisance parameter uncertainty is cleanly decoupled from the target parameter.

How does Fisher Information behave when we change coordinates? Let $\boldsymbol{\psi} = \mathbf{g}(\boldsymbol{\theta})$ be a smooth, invertible, one-to-one reparameterization with non-singular Jacobian matrix $\mathbf{J} = \frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\psi}}$.

By the chain rule of multivariable calculus:

In vector notation:

Taking the covariance matrix of both sides:

Inverting both sides:

This matches the asymptotic covariance matrix obtained from the Delta Method! Fisher Information transforms covariantly as a tensor metric of rank 2, exactly as in Riemannian differential geometry.

Variance-Stabilizing Transformations via Information Geometry

A classic problem in statistical methodology is finding a transformation $g(\theta)$ such that the asymptotic variance of $g(\hat{\theta})$ is completely constant and independent of the true parameter $\theta$. By the Delta Method:

To make this asymptotic variance equal to a constant $c^2$ independent of $\theta$, we set:

Integrating both sides yields the canonical Variance-Stabilizing Transformation:

Notice that $\sqrt{I_1(\theta)} \, d\theta$ is precisely the infinitesimal line element $ds$ of the Fisher Information Riemannian metric! Canonical examples derived directly from this information integral include:

1. Poisson Rate $\lambda$: $I_1(\lambda) = \frac{1}{\lambda}$. Then $g(\lambda) = \int \lambda^{-1/2} \, d\lambda = 2\sqrt{\lambda}$. Thus the square-root transform $\sqrt{X}$ has asymptotic variance $\frac{1}{4}$ independent of $\lambda$.

2. Binomial Proportion $p$: $I_1(p) = \frac{1}{p(1-p)}$. Then $g(p) = \int \frac{1}{\sqrt{p(1-p)}} \, dp = 2 \arcsin(\sqrt{p})$. Thus the angular transform $\arcsin(\sqrt{\hat{p}})$ stabilizes variance to $\frac{1}{4n}$.

3. Bivariate Normal Correlation $\rho$: $I_1(\rho) = \frac{1}{(1 - \rho^2)^2}$. Then $g(\rho) = \int \frac{1}{1 - \rho^2} \, d\rho = \frac{1}{2} \ln\left( \frac{1 + \rho}{1 - \rho} \right) = \text{arctanh}(\rho)$. This is Fisher's celebrated $z$-transformation, which stabilizes the sampling variance of Pearson's $r$ to $\frac{1}{n - 3}$.

Differential Geometry of the Fisher Information Metric

In 1945, C. R. Rao made the groundbreaking observation that the Fisher Information matrix acts as a Riemannian metric tensor on the manifold of probability distributions. Let $\mathcal{M} = \{P_\theta : \theta \in \Theta\}$ be a family of probability measures satisfying regularity conditions. The tangent space at $\theta$ is spanned by the score vectors $\frac{\partial \ln f}{\partial \theta_i}$. The Riemannian metric tensor is given by:

The geodesic distance between two probability measures $P_{\theta_1}$ and $P_{\theta_2}$ along the statistical manifold is the Rao Distance:

For Gaussian location-scale distributions $\mathcal{N}(\mu, \sigma^2)$, the Fisher Information metric is:

This is the Poincaré upper half-plane metric of hyperbolic geometry with constant negative curvature $K = -1/2$! This geometric insight reveals that statistical distinguishability between distributions behaves like hyperbolic distance in non-Euclidean geometry: distributions with small $\sigma$ are far apart even for tiny differences in $\mu$, whereas distributions with large $\sigma$ are tightly clustered together.

When the Cramér–Rao bound is strictly unattainable because the target function $\tau(\theta)$ is non-linear or the density does not admit a linear score, can we establish tighter, higher-order lower bounds?

In 1946, A. Bhattacharyya developed a sequence of ascending lower bounds that generalizes the Cramér–Rao inequality by considering higher-order derivatives of the likelihood.

Definition of Bhattacharyya Bounds

Define the higher-order score functions:

Notice that $S^{(1)}(\theta; \mathbf{X}) = \frac{\partial \ln f}{\partial \theta}$ is the standard Fisher score. The Bhattacharyya information matrix $\mathbf{W} \in \mathbb{R}^{K \times K}$ has entries:

For an unbiased estimator $T(\mathbf{X})$ of $\tau(\theta)$, define the derivative vector:

The $K$-th order Bhattacharyya Lower Bound is given by the quadratic form:

For $K = 1$, $\mathbf{W} = I_n(\theta)$ and $\mathbf{v} = \tau'(\theta)$, which reproduces the classical Cramér–Rao bound. For $K = 2$:

Because the Bhattacharyya bounds incorporate higher-order moments of the likelihood curvature, they form a monotonically non-decreasing sequence:

In many classical models (such as estimating $\sigma^4$ or $\mu^2$ in Gaussian samples), the sequence of Bhattacharyya bounds converges precisely to the exact variance of the UMVUE.

The classical Cramér–Rao bound requires differentiability of the density with respect to $\theta$ and an invariant support $\mathcal{S}$. When these regularity conditions fail—such as for the Uniform distribution $\text{Uniform}(0, \theta)$ where the support $[0, \theta]$ depends explicitly on $\theta$—the CRLB is inapplicable.

To overcome this fundamental limitation, D. G. Chapman and H. Robbins (1951), building on work by J. M. Hammersley (1950), established an information bound that requires zero differentiability conditions.

Theorem: The Chapman–Robbins Inequality

Let $\mathcal{P} = \{P_\theta : \theta \in \Theta\}$ be a family of probability measures. Let $T(\mathbf{X})$ be an unbiased estimator of $\tau(\theta)$ with finite variance. Let $\theta, \theta + \Delta \in \Theta$ be two distinct parameter points such that the support of $P_{\theta+\Delta}$ is contained within the support of $P_\theta$. Then:

Derivation for the Uniform Distribution:

Let $X_1, \dots, X_n \sim \text{Uniform}(0, \theta)$. To ensure the support condition holds, choose $\Delta < 0$, so that $[0, \theta + \Delta] \subset [0, \theta]$. The joint density ratio is:

provided $x_{(n)} \le \theta + \Delta$, and $0$ otherwise. The expectation of the squared ratio is:

Subtracting $1$:

For estimating $\tau(\theta) = \theta$, the numerator is $\Delta^2$. Setting $\epsilon = -\Delta / \theta \in (0, 1)$:

Using Taylor expansion $(1 - \epsilon)^{-n} - 1 = n\epsilon + \frac{n(n+1)}{2} \epsilon^2 + \mathcal{O}(\epsilon^3)$ and optimizing over $\epsilon$, the supremum evaluates to:

Remarkably, this bound exactly equals the variance of the UMVUE $T^* = \frac{n+1}{n} X_{(n)}$! The Chapman–Robbins inequality provides a non-trivial, sharp lower bound of order $\mathcal{O}(1/n^2)$ where classical Cramér–Rao theory fails completely.

In survival analysis, reliability engineering, and econometrics, data are frequently observed through imperfect observation windows:

Mathematical Derivation for Type I Right Censoring

Let $X_1, \dots, X_n$ be i.i.d. failure times from an Exponential distribution $\text{Exp}(\theta)$ with density $f(x; \theta) = \theta e^{-\theta x}$ and survival function $S(x; \theta) = e^{-\theta x}$. The experiment terminates at a fixed censoring threshold $C > 0$. For each unit $i$, we observe:

The likelihood contribution of an observation is:

The log-likelihood for the sample is:

where $d = \sum_{i=1}^n \delta_i$ is the random number of observed failures. Differentiating with respect to $\theta$:

Taking the negative expectation:

Because each $\delta_i \sim \text{Bernoulli}(p)$ with probability of failure $p = \mathbb{P}(X_i \le C) = 1 - e^{-\theta C}$, the expected number of failures is $\mathbb{E}[d] = n(1 - e^{-\theta C})$. Thus:

The Fisher Information under right censoring is directly proportional to the probability of observing an event! The quantity $e^{-\theta C} I_n^{\text{uncens}}(\theta)$ represents the exact information lost due to censoring.

Fisher Information in Grouped and Quantized Data

In survey sampling and physical telemetry, continuous signals $X$ are frequently discretized into $K$ disjoint intervals $A_k = (a_{k-1}, a_k]$ with probabilities:

For a sample of size $n$, the grouped observation is a count vector $(N_1, \dots, N_K) \sim \text{Multinomial}(n, p_1(\theta), \dots, p_K(\theta))$. The Fisher Information contained in the grouped data is:

Now, apply the Cauchy–Schwarz inequality to each interval integral:

Dividing both sides by $\int_{a_{k-1}}^{a_k} f(x; \theta) \, dx = p_k(\theta)$:

Summing across all $K$ intervals:

This proves that grouping or quantizing continuous measurements strictly destroys Fisher Information! The information gap vanishes only in the asymptotic continuum limit as the maximum bin mesh $\max_k |a_k - a_{k-1}| \to 0$.

Structural Comparison of Multiparameter Regularity Conditions

In multiparameter statistical estimation, establishing the attainability of the Cramér–Rao bound requires careful verification of regularity conditions over open subsets of Euclidean parameter spaces. When the parameter vector $\boldsymbol{\theta} = (\theta_1, \theta_2, \dots, \theta_k)^T$ lies in an open parameter manifold $\Theta \subseteq \mathbb{R}^k$, the Fisher information matrix is defined by the expected outer product of score vectors:

To guarantee that the inverse information matrix $\mathbf{I}(\boldsymbol{\theta})^{-1}$ exists and serves as a strict lower bound for the dispersion matrix of any unbiased vector estimator $\mathbf{T}(\mathbf{X})$, three fundamental conditions must hold simultaneously:

1. The support of the random vector $\mathbf{X}$, denoted by $\mathcal{X}_{\boldsymbol{\theta}} = \{\mathbf{x} : f(\mathbf{x}; \boldsymbol{\theta}) > 0\}$, must be strictly independent of all coordinates of $\boldsymbol{\theta}$. When the boundaries of the support depend on parameters (as in truncated distributions, shifted Pareto, or two-parameter uniform distributions), the classical score function exhibits delta-distribution singularities at the boundary, rendering the classical interchange of differentiation and integration invalid.
2. The partial derivatives $\frac{\partial f(\mathbf{x}; \boldsymbol{\theta})}{\partial \theta_j}$ must exist for almost all $\mathbf{x}$ with respect to the dominating measure, and must be dominated by Lebesgue-integrable functions over compact parameter neighborhoods. This domination property guarantees uniform integrability and justifies the application of the dominated convergence theorem.
3. The Fisher information matrix $\mathbf{I}(\boldsymbol{\theta})$ must be strictly positive definite everywhere on $\Theta$. Positive definiteness ensures that no linear combination of score components is almost surely zero, which is mathematically equivalent to the linear independence of the score coordinates and the absence of redundant or unidentifiable parameter directions.

When these conditions are met, the Chapman–Robbins inequality provides a non-differential lower bound that converges smoothly to the Cramér–Rao lower bound as the perturbation parameter approaches zero. This unifies differential and difference-quotient approaches to information theory.