Exponential Families of Distributions: Natural Parameters, Completeness, and Information Bounds

In the vast landscape of parametric probability models, one structural family stands out above all others for its exceptional analytical elegance, geometric regularity, and computational tractability: the Exponential Family of distributions. First identified systematically by Georges Darmois, Bernard Koopman, and E. J. G. Pitman in the mid-1930s, the exponential family encompasses almost all standard classical distributions—including the Gaussian, Bernoulli, Binomial, Poisson, Geometric, Negative Binomial, Multinomial, Exponential, Gamma, Beta, and von Mises distributions.

The General Representation

Let $(\mathcal{X}, \mathcal{A}, \mu)$ be a measurable space equipped with a $\sigma$-finite reference measure $\mu$. A parametric family of probability distributions $\mathcal{P} = \{P_{\boldsymbol{\theta}} : \boldsymbol{\theta} \in \Theta\}$ dominated by $\mu$ is said to be a $k$-parameter Exponential Family if the generalized probability density function (or probability mass function) can be expressed in the factored exponential form:

where:

1. $T_1(\mathbf{x}), T_2(\mathbf{x}), \dots, T_k(\mathbf{x})$ are real-valued Borel measurable functions defined on the sample space $\mathcal{X}$, termed the sufficient statistics.
2. $c_1(\boldsymbol{\theta}), c_2(\boldsymbol{\theta}), \dots, c_k(\boldsymbol{\theta})$ are real-valued functions of the parameter vector $\boldsymbol{\theta} \in \Theta$.
3. $d(\boldsymbol{\theta})$ is a real-valued scalar function ensuring that the density integrates to unity over $\mathcal{X}$.
4. $S(\mathbf{x})$ is a real-valued measurable function of the data vector alone.
5. $\mathcal{X}_0 \subseteq \mathcal{X}$ is the support of the distribution, which must be strictly invariant with respect to $\boldsymbol{\theta}$.

Equivalently, setting $h(\mathbf{x}) = \exp(S(\mathbf{x})) \mathbf{1}_{\mathcal{X}_0}(\mathbf{x})$ and $C(\boldsymbol{\theta}) = \exp(-d(\boldsymbol{\theta}))$, the family can be written in multiplicative standard form:

Notice that by the Fisher-Neyman Factorization Theorem (Unit 5), the $k$-dimensional vector:

is immediately and automatically a sufficient statistic for the parameter vector $\boldsymbol{\theta}$.

The Pitman-Koopman-Darmois Theorem: The Characterization of Sufficiency

A fundamental question that occupied mathematical statisticians in the 1930s was whether families of distributions possessing fixed-dimension sufficient statistics were rare curiosities or formed a well-defined mathematical class. In independent papers published between 1935 and 1936, Georges Darmois in France, Bernard Koopman in the United States, and E. J. G. Pitman in the United Kingdom resolved this problem completely.

Theorem (Pitman-Koopman-Darmois Theorem): Let $X_1, X_2, \dots, X_n$ be an independent and identically distributed random sample from a family of continuous distributions $\mathcal{P} = \{P_\theta : \theta \in \Theta\}$ with common support $\mathcal{X}_0$ that does not depend on the parameter $\theta$. Assume that the probability density function $f(x; \theta)$ is continuously differentiable with respect to $x$. If there exists a sufficient statistic $\mathbf{T}(X_1, \dots, X_n)$ whose dimension $k$ remains strictly bounded as the sample size $n$ approaches infinity ($k \le K < \infty$ for all $n \ge K$), then $\mathcal{P}$ is necessarily an Exponential Family.

The implications of this theorem are profound for computational and mathematical statistics:

1. Universal Dimension Reduction: If a statistical model does not belong to the exponential family, the dimension of the minimal sufficient statistic typically grows linearly with sample size ($k = n$). For example, for the Cauchy distribution $\text{Cauchy}(\theta, 1)$ or the Student's $t$-distribution, no data reduction is possible without losing Fisher information: the minimal sufficient statistic is the entire order statistic $(X_{(1)}, \dots, X_{(n)})$. To compute likelihoods, one must retain all $n$ raw data points in memory.

2. Support Invariance is Crucial: If the support depends upon the parameter $\theta$, non-exponential families can admit finite-dimensional sufficient statistics. For instance, in the Uniform distribution $\text{Uniform}(0, \theta)$, the support is $[0, \theta]$, and the single scalar order statistic $X_{(n)} = \max(X_1, \dots, X_n)$ is sufficient for all $n$. However, because the support boundary shifts with $\theta$, the family violates the support invariance axiom and falls outside the exponential family framework.

The Canonical (Natural) Representation

The standard representation often obscures the underlying convex geometry because the functions $c_j(\boldsymbol{\theta})$ may be non-linear. To reveal the natural geometry of the family, we execute a reparameterization. Define the natural (or canonical) parameters:

Grouping them into the natural parameter vector $\boldsymbol{\eta} = (\eta_1, \eta_2, \dots, \eta_k)^T \in \mathbb{R}^k$, and the sufficient statistics into $\mathbf{T}(\mathbf{x}) = (T_1(\mathbf{x}), \dots, T_k(\mathbf{x}))^T$, the linear combination in the exponent becomes the standard Euclidean inner product:

The normalizing constant $C(\boldsymbol{\theta})$ is determined by the total probability axiom:

Factoring out the term depending solely on $\boldsymbol{\eta}$, we define the log-partition function (also known as the cumulant function or free energy):

This yields the celebrated Canonical (Natural) Exponential Family Representation:

where $h(\mathbf{x}) = e^{S(\mathbf{x})} \mathbf{1}_{\mathcal{X}_0}(\mathbf{x})$ is the base reference measure density.

The set of all natural parameter vectors for which the density can be normalized into a legitimate probability measure is called the natural parameter space.

Definition (Natural Parameter Space): The natural parameter space $\Xi \subseteq \mathbb{R}^k$ of an exponential family is defined as the set of all vectors $\boldsymbol{\eta} \in \mathbb{R}^k$ for which the normalizing integral is strictly finite:

A remarkable mathematical property of the exponential family is that its natural parameter space is universally convex.

Theorem (Convexity of the Natural Parameter Space): The natural parameter space $\Xi$ is a convex subset of $\mathbb{R}^k$. Furthermore, the log-partition function $A(\boldsymbol{\eta})$ is a strictly convex function on $\Xi$ (provided the statistics $T_1, \dots, T_k$ are affinely independent).

Proof: Let $\boldsymbol{\eta}_1, \boldsymbol{\eta}_2 \in \Xi$, and let $\alpha \in (0, 1)$. Define the convex combination $\boldsymbol{\eta}_\alpha = \alpha \boldsymbol{\eta}_1 + (1 - \alpha) \boldsymbol{\eta}_2$. We must show that $\boldsymbol{\eta}_\alpha \in \Xi$. Consider the normalizing integral for $\boldsymbol{\eta}_\alpha$:

Express $h(\mathbf{x}) = [h(\mathbf{x})]^\alpha [h(\mathbf{x})]^{1 - \alpha}$ and apply Hölder's Inequality with conjugate exponents $p = 1/\alpha > 1$ and $q = 1/(1 - \alpha) > 1$ (satisfying $1/p + 1/q = \alpha + (1 - \alpha) = 1$):

Because $\boldsymbol{\eta}_1, \boldsymbol{\eta}_2 \in \Xi$, both $M(\boldsymbol{\eta}_1) < \infty$ and $M(\boldsymbol{\eta}_2) < \infty$. Therefore, their weighted geometric product is strictly finite:

Thus, $\Xi$ is a convex set.

Taking natural logarithms across Hölder's inequality:

Recalling that $A(\boldsymbol{\eta}) = \ln M(\boldsymbol{\eta})$:

This is precisely the definition of a convex function. By Hölder's equality condition, equality holds if and only if $\exp(\boldsymbol{\eta}_1^T \mathbf{T}(\mathbf{x})) = c \exp(\boldsymbol{\eta}_2^T \mathbf{T}(\mathbf{x}))$ almost surely for some constant $c > 0$, which implies $(\boldsymbol{\eta}_1 - \boldsymbol{\eta}_2)^T \mathbf{T}(\mathbf{x}) = \ln c$. If the coordinate functions $T_1(\mathbf{x}), \dots, T_k(\mathbf{x})$ and the constant 1 are linearly independent with respect to the reference measure $\mu$ (affinely independent), then $\boldsymbol{\eta}_1 = \boldsymbol{\eta}_2$. Therefore, $A(\boldsymbol{\eta})$ is strictly convex on $\Xi$. $\blacksquare$

Regular Versus Curved Exponential Families

A $k$-parameter exponential family is classified based on the relationship between its parameter dimension and its sufficient statistics:

1. Full-Rank (Regular) Exponential Family: The natural parameter space $\Xi$ has a non-empty interior in $\mathbb{R}^k$ (contains an open $k$-dimensional rectangle or ball), and the dimension of $\boldsymbol{\theta}$ equals the dimension of the sufficient statistic: $\dim(\boldsymbol{\theta}) = k$. In full-rank families, classical asymptotic and finite-sample properties hold with maximal power.

2. Curved Exponential Family: The original parameter space $\Theta \subseteq \mathbb{R}^d$ has dimension $d$ strictly smaller than the dimension $k$ of the minimal sufficient statistic: $d < k$. The mapping $\boldsymbol{\theta} \mapsto \boldsymbol{\eta}(\boldsymbol{\theta})$ defines a $d$-dimensional curved manifold embedded within the $k$-dimensional natural parameter space $\mathbb{R}^k$. A famous example is the normal distribution $\mathcal{N}(\mu, \mu^2)$ where the variance equals the square of the mean ($d = 1$, but $k = 2$ with $\mathbf{T} = (\sum X_i, \sum X_i^2)$). Bradley Efron (1975) developed the concept of statistical curvature to quantify the information loss that occurs in curved exponential families relative to full-rank families.

In classical calculus, evaluating the moments of a multivariate distribution requires executing complicated multidimensional integrals. In the exponential family, the log-partition function $A(\boldsymbol{\eta})$ acts as a master cumulant generating potential: differentiating $A(\boldsymbol{\eta})$ algebraically generates the moments and cumulants of the sufficient statistics without performing any integration!

The Gradient of $A(\boldsymbol{\eta})$ Generates the Mean Vector

Recall that:

When $\boldsymbol{\eta}$ resides in the interior of the natural parameter space $\Xi^\circ$, the integrand is dominated by integrable functions in a neighborhood of $\boldsymbol{\eta}$. By the Leibniz Integral Rule and the Dominated Convergence Theorem, we can differentiate under the integral sign with respect to the $j$-th natural coordinate $\eta_j$:

Using the chain rule on the left side: $\frac{\partial}{\partial \eta_j} \exp(A(\boldsymbol{\eta})) = \exp(A(\boldsymbol{\eta})) \frac{\partial A(\boldsymbol{\eta})}{\partial \eta_j}$. Dividing through by $\exp(A(\boldsymbol{\eta}))$:

Notice that the right-hand side is precisely the definition of the mathematical expectation of $T_j(\mathbf{X})$ under parameter $\boldsymbol{\eta}$! Therefore:

In vector gradient notation, the mean vector of the sufficient statistics is given by the gradient of the log-partition function:

The Hessian Matrix of $A(\boldsymbol{\eta})$ Generates the Covariance Matrix

Differentiating a second time under the integral sign with respect to $\eta_m$:

Applying the product rule to the exponential term:

Rewriting in terms of mathematical expectations:

Since $\mathbb{E}_{\boldsymbol{\eta}}\left[ \mathbb{E}_{\boldsymbol{\eta}}[T_j(\mathbf{X})] (T_m(\mathbf{X}) - \mathbb{E}_{\boldsymbol{\eta}}[T_m(\mathbf{X})]) \right] = 0$, this is identically equal to the covariance:

In matrix Hessian notation, the covariance matrix of the sufficient statistics is given by the Hessian of the log-partition function:

Because a covariance matrix is always positive semi-definite (and strictly positive-definite if the statistics are affinely independent), this provides an independent proof that the Hessian $\nabla^2 A(\boldsymbol{\eta}) \succ 0$ is positive-definite, confirming that $A(\boldsymbol{\eta})$ is strictly convex on $\Xi^\circ$.

Fisher Information in Canonical Exponential Families

In Unit 1, Fisher Information was defined as the variance of the score function. In a canonical exponential family, the log-likelihood is:

The score vector is the gradient with respect to $\boldsymbol{\eta}$:

Notice that the score vector is simply the centered sufficient statistic! The Fisher Information matrix is the covariance of the score vector:

Thus, in canonical exponential families, the Fisher Information matrix is identically equal to the Hessian of the log-partition function:

This fundamental identity connects information geometry directly with convex analysis: the Riemannian metric tensor of the statistical manifold is precisely the Hessian of the potential function $A(\boldsymbol{\eta})$.

The moment generating function of the sufficient statistic $\mathbf{T}(\mathbf{X})$ can also be evaluated directly from the log-partition function without integrating.

Let $\mathbf{t} \in \mathbb{R}^k$. The MGF of $\mathbf{T}(\mathbf{X})$ under parameter $\boldsymbol{\eta}$ is:

Combine the exponential terms:

Recognizing the integral as $\exp(A(\boldsymbol{\eta} + \mathbf{t}))$:

Taking natural logarithms, the cumulant generating function of $\mathbf{T}(\mathbf{X})$ is:

This proves that $A(\boldsymbol{\eta})$ is the universal cumulant generating function for the sufficient statistics: evaluating the $r$-th mixed partial derivatives of $A(\boldsymbol{\eta})$ yields the exact $r$-th joint cumulants of $(T_1, \dots, T_k)$ across all orders!

In Unit 5, we saw that completeness is the crucial condition required by the Lehmann-Scheffé theorem to guarantee the uniqueness of UMVUEs and by Basu's theorem to prove statistical independence. In full-rank exponential families, completeness is guaranteed automatically.

Theorem (Completeness of Full-Rank Exponential Families): Let $\mathcal{P} = \{f(\mathbf{x}; \boldsymbol{\eta}) : \boldsymbol{\eta} \in \Xi\}$ be a $k$-parameter canonical exponential family. If the natural parameter space $\Xi$ contains a non-empty open rectangle:

then the natural sufficient statistic:

is a complete statistic for the family $\mathcal{P}$.

Proof Strategy via Fourier Inversion: Suppose $g(\mathbf{t})$ is any Borel measurable function of $\mathbf{T}$ such that:

Let $\nu$ denote the push-forward measure of $h(\mathbf{x}) \, d\mu(\mathbf{x})$ under $\mathbf{T}$. The expectation can be written as an integral over the range space $\mathbb{R}^k$:

Multiplying by $\exp(A(\boldsymbol{\eta})) > 0$:

Decompose $g(\mathbf{t}) = g^+(\mathbf{t}) - g^-(\mathbf{t})$ into its positive and negative parts:

Fix any point $\boldsymbol{\eta}_0 \in \mathcal{O}$. Define two positive finite measures on $(\mathbb{R}^k, \mathcal{B}(\mathbb{R}^k))$:

The integral equation states that for all $\mathbf{s} \in \mathbb{R}^k$ such that $\boldsymbol{\eta}_0 + \mathbf{s} \in \mathcal{O}$:

Because $\boldsymbol{\eta}_0$ is in the interior $\mathcal{O}$, this equality holds on an open neighborhood of $\mathbf{s} = \mathbf{0}$. Analytically continuing this relation from real arguments $\mathbf{s}$ to purely imaginary arguments $i\mathbf{u}$ (where $\mathbf{u} \in \mathbb{R}^k$ and $i = \sqrt{-1}$), the identity theorem for complex analytic functions guarantees that:

Notice that the left and right sides are the characteristic functions (multidimensional Fourier transforms) of the measures $\nu_1$ and $\nu_2$! By the uniqueness theorem of Fourier transforms (Lévy's Inversion Theorem), two measures having identical characteristic functions must be identical:

Equating their Radon-Nikodym derivatives:

Dividing by $\exp(\boldsymbol{\eta}_0^T \mathbf{t}) > 0$:

Therefore:

This proves that $\mathbf{T}(\mathbf{X})$ is a complete statistic. $\blacksquare$

This theorem guarantees that for Gaussian, Poisson, Gamma, Beta, and Binomial distributions, the natural sufficient statistics are unconditionally complete. Every unbiased estimator based on them is automatically the unique UMVUE.

The natural parameter vector $\boldsymbol{\eta} \in \Xi$ provides an extrinsic mathematical representation of the distribution. An alternative, deeply intuitive intrinsic parameterization is provided by the expectation parameters (or mean-value parameters):

Because $A(\boldsymbol{\eta})$ is strictly convex on $\Xi$, the gradient mapping:

is a smooth, one-to-one diffeomorphism from the interior of the natural parameter space $\Xi^\circ$ onto the interior of the marginal polytope (the convex hull of the support of $\mathbf{T}$):

Legendre-Fenchel Convex Duality

The relationship between the natural parameters $\boldsymbol{\eta}$ and expectation parameters $\boldsymbol{\mu}$ is governed by Legendre-Fenchel convex duality. The convex conjugate (Legendre transform) of the log-partition function $A(\boldsymbol{\eta})$ is defined by:

The supremum is achieved at the unique point $\boldsymbol{\eta}$ where the gradient equals $\boldsymbol{\mu}$: $\nabla A(\boldsymbol{\eta}) = \boldsymbol{\mu}$. Substituting this optimal $\boldsymbol{\eta}(\boldsymbol{\mu})$:

The dual function $A^*(\boldsymbol{\mu})$ has a profound physical interpretation: it is precisely the negative Shannon entropy (or Boltzmann-Gibbs entropy) of the distribution! Furthermore, the gradient of the dual function inverts the mapping:

This establishes a perfect symmetric duality between natural parameters and expectation parameters:

• $\nabla A(\boldsymbol{\eta}) = \boldsymbol{\mu}$ maps natural parameters to expected moments.
• $\nabla A^*(\boldsymbol{\mu}) = \boldsymbol{\eta}$ maps expected moments back to natural parameters.

Maximum Entropy Characterization

Exponential families arise naturally as the unique solutions to constrained entropy maximization problems (the Principle of Maximum Entropy, E. T. Jaynes, 1957).

Theorem (Maximum Entropy Law): Among all probability distributions on $\mathcal{X}$ satisfying specified moment constraints:

the unique probability distribution that maximizes the Shannon differential entropy:

is the Exponential Family distribution $f(\mathbf{x}; \boldsymbol{\eta}) = \exp\left( \boldsymbol{\eta}^T \mathbf{T}(\mathbf{x}) - A(\boldsymbol{\eta}) \right)$, where $\boldsymbol{\eta} = \nabla A^*(\boldsymbol{\mu})$ is chosen to satisfy the moment constraints.

This proves that exponential families are the most unbiased, least informative, and maximally neutral distributions compatible with observed empirical moments.

In Bayesian inference, an analyst specifies a prior probability distribution $\pi(\boldsymbol{\theta})$ over the parameter space $\Theta$. Upon observing data $\mathbf{x}$, Bayes' theorem updates the prior to the posterior distribution:

A family of prior distributions is called conjugate to a likelihood model if the posterior distribution belongs to the exact same parametric family as the prior.

In 1979, Persi Diaconis and Donald Ylvisaker proved that canonical exponential families possess natural conjugate priors of exceptional simplicity.

Let $f(\mathbf{x}; \boldsymbol{\eta}) = \exp\left( \boldsymbol{\eta}^T \mathbf{T}(\mathbf{x}) - A(\boldsymbol{\eta}) \right) h(\mathbf{x})$ be a canonical exponential family. Define the prior distribution over the natural parameter vector $\boldsymbol{\eta} \in \Xi$ by:

where $n_0 > 0$ represents the prior "effective sample size", $\mathbf{t}_0 \in \mathbb{R}^k$ represents prior "pseudo-observations", and $B(n_0, \mathbf{t}_0)$ is the normalizing constant.

Now, observe a sample $\mathbf{X}_1, \dots, \mathbf{X}_n$ with observed sufficient statistic $\mathbf{T}_{\text{obs}} = \sum_{i=1}^n \mathbf{T}(\mathbf{X}_i)$. The likelihood of the sample is:

Multiplying the prior by the likelihood:

Combine the linear exponents:

Notice that the posterior distribution belongs to the exact same parametric family with updated hyperparameters:

Bayesian updating in exponential families reduces to simple arithmetic addition:

• Prior sample size $n_0$ increases by the observed sample size $n$.
• Prior pseudo-statistic $\mathbf{t}_0$ increases by the observed sufficient statistic $\mathbf{T}_{\text{obs}}$.

Examples include the Beta-Binomial, Dirichlet-Multinomial, Gamma-Poisson, and Normal-Inverse-Gamma conjugate updating models.

In point estimation, a fundamental lower bound limits the precision achievable by any unbiased estimator.

Theorem (Cramér-Rao Inequality): Let $W(\mathbf{X})$ be an unbiased estimator of a scalar parameter function $\tau(\theta)$ under a regular parametric family with Fisher Information $I(\theta)$. Then the variance of $W$ is bounded below by:

Characterization of Equality Attainment

When is the Cramér-Rao Lower Bound attained with exact equality? By the Cauchy-Schwarz inequality used in the proof of the Cramér-Rao inequality, equality holds if and only if the estimator $W(\mathbf{X})$ is a linear function of the score function $S(\theta; \mathbf{X}) = \frac{\partial}{\partial \theta} \ln f(\mathbf{X}; \theta)$:

for some function $k(\theta) \ne 0$. Integrating this differential equation with respect to $\theta$:

Integrating with respect to $\theta$:

Exponentiating both sides:

where $c(\theta) = \int \frac{1}{k(\theta)} \, d\theta$ and $d(\theta) = \int \frac{\tau(\theta)}{k(\theta)} \, d\theta$.

This establishes a definitive theorem:

Theorem (Cramér-Rao Efficiency Attainment): The Cramér-Rao Lower Bound is achieved by an unbiased estimator if and only if the underlying family of distributions is an Exponential Family, and the parameter function being estimated is $\tau(\theta) = \mathbb{E}_\theta[W(\mathbf{X})]$, where $W(\mathbf{X})$ is the natural sufficient statistic!

No estimator outside the Exponential Family can ever achieve the Cramér-Rao Lower Bound across its entire parameter space.

In modern mathematical statistics, a parametric family of distributions $\mathcal{S} = \{P_{\boldsymbol{\theta}} : \boldsymbol{\theta} \in \Theta\}$ is conceptualized as a smooth differential manifold—termed a statistical manifold—where each point on the manifold represents an entire probability distribution. The pioneer of this geometric synthesis, Shun-ichi Amari (1985), demonstrated that canonical exponential families possess an extraordinary geometric structure known as dual flatness.

The Fisher-Rao Riemannian Metric

In classical differential geometry, a manifold is equipped with a Riemannian metric tensor $g_{ij}(\boldsymbol{\theta})$ that defines inner products on the tangent spaces, measuring infinitesimal distances between neighboring points. C. R. Rao (1945) proved that the unique, geometrically invariant Riemannian metric on a statistical manifold is given by the Fisher Information matrix:

The geodesic distance between two probability distributions $P_{\boldsymbol{\theta}_1}$ and $P_{\boldsymbol{\theta}_2}$ under this metric is known as the Fisher-Rao distance:

where the infimum is taken over all smooth paths $\gamma(t)$ connecting $\boldsymbol{\theta}_1$ to $\boldsymbol{\theta}_2$.

Dual Affine Connections and Flatness

In a canonical exponential family, the Fisher Information matrix is the Hessian of the log-partition function: $\mathbf{I}(\boldsymbol{\eta}) = \nabla^2 A(\boldsymbol{\eta})$. Amari introduced a one-parameter family of affine connections $\nabla^{(\alpha)}$ indexed by $\alpha \in \mathbb{R}$:

• The connection $\nabla^{(1)}$ is termed the exponential connection ($e$-connection).
• The connection $\nabla^{(-1)}$ is termed the mixture connection ($m$-connection).

A statistical manifold is said to be $e$-flat if its Riemann curvature tensor with respect to the $e$-connection vanishes identically: $R^{(e)} = 0$. A manifold is $m$-flat if $R^{(m)} = 0$.

In a full-rank exponential family:

1. The natural parameterization $\boldsymbol{\eta}$ forms an affine coordinate system with respect to the $e$-connection. The manifold is strictly $e$-flat. Geodesics under the $e$-connection are straight lines in the natural parameter space: $\boldsymbol{\eta}(t) = (1 - t)\boldsymbol{\eta}_1 + t \boldsymbol{\eta}_2$.
2. The expectation parameterization $\boldsymbol{\mu}$ forms an affine coordinate system with respect to the $m$-connection. The manifold is strictly $m$-flat. Geodesics under the $m$-connection are straight lines in the expectation parameter space: $\boldsymbol{\mu}(t) = (1 - t)\boldsymbol{\mu}_1 + t \boldsymbol{\mu}_2$.

Because the manifold is simultaneously $e$-flat and $m$-flat with respect to mutually dual affine connections, canonical exponential families are dually flat Riemannian spaces.

While the Fisher-Rao metric measures infinitesimal local distances, the global separation between two probability distributions $P$ and $Q$ is quantified by the Kullback-Leibler (KL) divergence (or relative entropy):

By Jensen's Inequality, $D_{\text{KL}}(P \parallel Q) \ge 0$, with equality if and only if $P = Q$ almost surely.

Representation via the Log-Partition Function

Consider two members of a canonical exponential family with natural parameters $\boldsymbol{\eta}_1$ and $\boldsymbol{\eta}_2$:

The log-density ratio is:

Taking the expectation under $P_{\boldsymbol{\eta}_1}$:

Recalling that $\boldsymbol{\mu}_1 = \mathbb{E}_{\boldsymbol{\eta}_1}[\mathbf{T}(\mathbf{X})] = \nabla A(\boldsymbol{\eta}_1)$, we obtain:

Notice that the right-hand side is precisely the definition of the Bregman divergence $B_A(\boldsymbol{\eta}_2, \boldsymbol{\eta}_1)$ associated with the strictly convex function $A$:

The KL divergence represents the vertical discrepancy between the convex surface $A(\boldsymbol{\eta}_2)$ and its tangent plane linearized at $\boldsymbol{\eta}_1$!

The Generalized Pythagorean Theorem

In Euclidean geometry, the Pythagorean theorem states that if two vectors $\mathbf{a} - \mathbf{b}$ and $\mathbf{b} - \mathbf{c}$ are orthogonal, then $\|\mathbf{a} - \mathbf{c}\|^2 = \|\mathbf{a} - \mathbf{b}\|^2 + \|\mathbf{b} - \mathbf{c}\|^2$. In information geometry, a remarkable generalized Pythagorean theorem governs dually flat exponential manifolds.

Theorem (Generalized Pythagorean Theorem): Let $P_{\boldsymbol{\eta}_1}$, $P_{\boldsymbol{\eta}_2}$, and $P_{\boldsymbol{\eta}_3}$ be three distributions on an exponential manifold. If the $e$-geodesic connecting $\boldsymbol{\eta}_1$ to $\boldsymbol{\eta}_2$ is orthogonal (under the Fisher metric) to the $m$-geodesic connecting $\boldsymbol{\eta}_2$ to $\boldsymbol{\eta}_3$:

then the Kullback-Leibler divergence decomposes additively:

This Pythagorean decomposition provides the mathematical foundation for maximum likelihood estimation, projection onto linear exponential sub-families, and information projection (I-projection) in machine learning.

When the parameter of interest $\theta \in \Theta \subseteq \mathbb{R}$ is scalar, but the minimal sufficient statistic $\mathbf{T}(\mathbf{X}) \in \mathbb{R}^k$ has dimension $k > 1$, the model forms a curved exponential family. The mapping $\theta \mapsto \boldsymbol{\eta}(\theta)$ traces out a 1-dimensional curved curve within the $k$-dimensional natural parameter space $\mathbb{R}^k$.

Bradley Efron (1975) introduced the fundamental concept of statistical curvature to quantify how severely a curved exponential model deviates from a full-rank family.

Definition of Statistical Curvature

Let $\mathbf{T}(\mathbf{X})$ be the $k$-dimensional natural sufficient statistic with covariance matrix $\boldsymbol{\Sigma}_{\boldsymbol{\eta}} = \nabla^2 A(\boldsymbol{\eta}) = \mathbf{I}(\boldsymbol{\eta})$. Differentiating the natural parameter curve $\boldsymbol{\eta}(\theta)$ with respect to $\theta$:

• The velocity vector is $\dot{\boldsymbol{\eta}}(\theta) = \frac{d\boldsymbol{\eta}}{d\theta} \in \mathbb{R}^k$.
• The acceleration vector is $\ddot{\boldsymbol{\eta}}(\theta) = \frac{d^2\boldsymbol{\eta}}{d\theta^2} \in \mathbb{R}^k$.

The Fisher Information for the scalar parameter $\theta$ from a single observation is:

The statistical curvature $\gamma_\theta \ge 0$ is defined as the norm of the component of the acceleration vector $\ddot{\boldsymbol{\eta}}(\theta)$ that is orthogonal to the velocity vector $\dot{\boldsymbol{\eta}}(\theta)$ under the Fisher Riemannian metric:

Notice that if the parameterization $\boldsymbol{\eta}(\theta)$ is a straight line in natural coordinates ($\ddot{\boldsymbol{\eta}} \propto \dot{\boldsymbol{\eta}}$), the orthogonal component vanishes: $\gamma_\theta = 0$. Thus, a model has zero statistical curvature if and only if it is a full-rank (linear) exponential family.

Asymptotic Consequences of Statistical Curvature

Efron proved that statistical curvature governs the second-order asymptotic efficiency of the Maximum Likelihood Estimator (MLE) $\hat{\theta}_n$:

1. Asymptotic Variance Inflation: In large samples, while the first-order asymptotic variance of $\hat{\theta}_n$ achieves the Cramér-Rao bound $\frac{1}{n I(\theta)}$, the second-order expansion reveals:

The statistical curvature $\gamma_\theta^2$ directly quantifies the asymptotic variance inflation of the MLE!

2. Fisher Information Loss: In a sample of size $n$, the Fisher Information contained in the MLE $\hat{\theta}_n$ satisfies:

The quantity $\gamma_\theta^2 I(\theta)$ represents the permanent, irreversible loss of Fisher Information incurred by reducing the $k$-dimensional sufficient statistic $\mathbf{T}$ to the scalar MLE $\hat{\theta}_n$. To recover this lost information, an analyst must condition on an ancillary statistic measuring the distance from the data to the curved manifold.

Exponential families provide the natural structural domain for the Neyman-Pearson theory of optimal hypothesis testing.

Monotone Likelihood Ratio (MLR)

A 1-parameter family of distributions $\{f(x; \theta) : \theta \in \Theta \subseteq \mathbb{R}\}$ is said to possess a Monotone Likelihood Ratio (MLR) in a scalar statistic $T(X)$ if for any two parameter values $\theta_1 < \theta_2$, the likelihood ratio:

is a non-decreasing (monotone) function of $T(x)$.

Every 1-parameter canonical exponential family:

possesses a Monotone Likelihood Ratio in $T(X)$ if and only if the natural parameter $\eta(\theta)$ is a strictly increasing function of $\theta$.

The Karlin-Rubin Theorem

In 1956, Samuel Karlin and Herman Rubin proved that families possessing an MLR admit Uniformly Most Powerful (UMP) tests for one-sided hypotheses.

Theorem (Karlin-Rubin Theorem): Suppose $X$ has a distribution with an MLR in $T(X)$. Consider testing the one-sided null hypothesis:

Then for any significance level $\alpha \in (0, 1)$, the test that rejects $H_0$ if and only if $T(X) > c$:

where the critical value $c$ and randomization constant $\gamma \in [0, 1]$ are chosen such that $\mathbb{E}_{\theta_0}[\phi(X)] = \alpha$, is the Uniformly Most Powerful (UMP) test of size $\alpha$.

Furthermore, the power function $\beta(\theta) = \mathbb{E}_\theta[\phi(X)]$ is strictly increasing in $\theta$ for all $\theta$ where $\beta(\theta) < 1$.

The Karlin-Rubin Theorem establishes that for 1-parameter exponential families, one-sided hypothesis tests based on the natural sufficient statistic are unbeatable: no competing test can achieve greater statistical power against any alternative $\theta > \theta_0$.

In modern statistical practice and applied regression modeling, classical linear regression models are generalized to accommodate non-Gaussian response variables (such as binary binary classification, Poisson event counts, and Gamma survival durations) through the framework of Generalized Linear Models (GLMs), developed by John Nelder and Robert Wedderburn in 1972.

GLMs model the response variable $Y$ using an Exponential Dispersion Family:

where:

1. $\theta$ is the natural parameter of location.
2. $\phi > 0$ is the dispersion parameter (governing variance scaling). Typically, $a(\phi) = \phi / w$, where $w > 0$ is a known observation weight.
3. $b(\theta)$ is the log-partition function, termed the cumulant function.

Mean and Variance Functions in GLMs

Applying the moment-generating properties of exponential families (Section 6.3):

1. Expected Value:

2. Variance:

where $V(\mu) = b''(\theta(\mu))$ is the variance function, which characterizes how the variance of the response variable depends systematically on its expected value $\mu$.

Prominent examples of variance functions include:

• Gaussian: $b(\theta) = \theta^2 / 2 \implies V(\mu) = 1$ (constant variance, homoskedastic).
• Poisson: $b(\theta) = e^\theta \implies V(\mu) = \mu$ (variance equals mean).
• Binomial ($m$ trials): $b(\theta) = m \ln(1 + e^\theta) \implies V(\mu) = \mu(1 - \mu/m)$.
• Gamma: $b(\theta) = -\ln(-\theta) \implies V(\mu) = \mu^2$ (constant coefficient of variation).
• Inverse Gaussian: $b(\theta) = -\sqrt{-2\theta} \implies V(\mu) = \mu^3$.

The Link Function and Canonical Links

A GLM connects the expected value $\mu_i = \mathbb{E}[Y_i]$ to a linear predictor $\eta_i = \mathbf{x}_i^T \boldsymbol{\beta}$ through a monotonic, differentiable link function $g$:

The canonical link function is chosen such that the linear predictor directly equals the natural parameter:

Under canonical links:

• For Gaussian data: $g(\mu) = \mu$ (identity link).
• For Bernoulli data: $g(\mu) = \ln\left(\frac{\mu}{1 - \mu}\right)$ (logit link, yielding logistic regression).
• For Poisson data: $g(\mu) = \ln\mu$ (log link, yielding Poisson log-linear regression).
• For Gamma data: $g(\mu) = 1/\mu$ (reciprocal link).

Under canonical link functions, the sufficient statistics for the regression coefficients $\boldsymbol{\beta}$ are simply $\mathbf{X}^T \mathbf{y} = \sum_{i=1}^n \mathbf{x}_i y_i$, the log-likelihood is strictly concave in $\boldsymbol{\beta}$, and the Newton-Raphson algorithm coincides with Fisher's method of scoring (Iteratively Reweighted Least Squares / IRLS), guaranteeing global quadratic convergence to the unique maximum likelihood estimator.

In Section 6.7, we observed that canonical exponential families admit natural conjugate prior distributions that update algebraically. A foundational theorem by Persi Diaconis and Donald Ylvisaker (1979) establishes an even deeper structural property: the posterior expectation of the mean parameter is strictly linear in the sufficient statistic.

Let $\mathbf{X} \sim f(\mathbf{x}; \boldsymbol{\eta}) = \exp(\boldsymbol{\eta}^T \mathbf{T}(\mathbf{x}) - A(\boldsymbol{\eta})) h(\mathbf{x})$ be a canonical exponential family, and let $\boldsymbol{\mu}(\boldsymbol{\eta}) = \mathbb{E}_{\boldsymbol{\eta}}[\mathbf{T}(\mathbf{X})] = \nabla A(\boldsymbol{\eta})$ denote the expectation parameter. Consider the natural conjugate prior on $\boldsymbol{\eta}$:

where $n_0 > 0$ and $\mathbf{t}_0 / n_0 = \boldsymbol{\mu}_0 \in \mathcal{M}^\circ$ represents the prior expected value of $\boldsymbol{\mu}$.

Theorem (Diaconis-Ylvisaker Linear Posterior Expectation Theorem): Under the natural conjugate prior $\pi(\boldsymbol{\eta}; n_0, \mathbf{t}_0)$, the posterior expectation of the mean parameter $\boldsymbol{\mu} = \nabla A(\boldsymbol{\eta})$ given an observed sample $\mathbf{X}_1, \dots, \mathbf{X}_n$ with sample mean sufficient statistic $\bar{\mathbf{T}} = \frac{1}{n}\sum_{i=1}^n \mathbf{T}(\mathbf{X}_i)$ is a strictly convex linear combination of the prior mean $\boldsymbol{\mu}_0$ and the sample average $\bar{\mathbf{T}}$:

where the shrinkage weight is:

Proof: Let $\mathbf{t}_n = \mathbf{t}_0 + n \bar{\mathbf{T}}$ and $n^* = n_0 + n$. The posterior density of $\boldsymbol{\eta}$ is:

Consider the integral of the gradient $\nabla_{\boldsymbol{\eta}} \pi(\boldsymbol{\eta} \mid \text{data})$ over the interior of the natural parameter space $\Xi^\circ$:

Differentiating the posterior density:

By the divergence theorem, the integral of the gradient of a probability density vanishing on the boundary of $\Xi$ is identically zero:

Separating the two terms:

Since the posterior integrates to unity ($\int \pi \, d\boldsymbol{\eta} = 1$):

Dividing through by $n^* = n_0 + n$:

This proves that the Bayes estimator under quadratic loss is an exact convex weighted average of the prior mean and the sample evidence. $\blacksquare$

Diaconis and Ylvisaker further established the remarkable converse: under mild regularity conditions, if the posterior expectation of the mean parameter is linear in the sample mean for all sample sizes $n$, the underlying distribution must belong to the Exponential Family, and the prior must be the natural conjugate prior!

In algorithmic information theory (A. N. Kolmogorov, 1965) and Jorma Rissanen's Minimum Description Length (MDL) principle (1978), statistical modeling is viewed as data compression. The goal of science is to identify regularities in data that permit the shortest possible binary encoding of the observed experience.

The Two-Part Code Interpretation of Sufficiency

Consider transmitting an observed dataset $\mathbf{x} = (x_1, \dots, x_n)$ across a noiseless binary communications channel to a receiver. A two-part code transmits:

1. The Model (Hypothesis): A compressed binary description of the estimated parameter or sufficient statistic $T(\mathbf{x})$.

2. The Residual Data given the Model: The index of the specific dataset $\mathbf{x}$ within the conditional equivalence class $T^{-1}(\{T(\mathbf{x})\})$.

The total code length (in Shannon bits) is:

By Shannon's Source Coding Theorem, the ideal code length for transmitting $\mathbf{x}$ under a probability model $P_{\boldsymbol{\theta}}$ is:

Now, observe what happens when $T$ is a sufficient statistic: By definition of sufficiency, the conditional distribution $P(\mathbf{X} = \mathbf{x} \mid T(\mathbf{X}) = T(\mathbf{x}))$ contains zero dependence on $\boldsymbol{\theta}$. Therefore, the second part of the code—the residual data given the sufficient statistic—has a code length:

which is completely universal and independent of $\boldsymbol{\theta}$! All the regularities and structure in the data that can be compressed by learning the parameter $\boldsymbol{\theta}$ are entirely captured by the first part: $-\log_2 P_{\boldsymbol{\theta}}(T(\mathbf{X}) = T(\mathbf{x}))$. The residual is pure, incompressible random noise (uniform across the partition cell).

This algorithmic equivalence demonstrates that a sufficient statistic in mathematical statistics is identically equal to a minimal sufficient description in computer science, bridging classical likelihood theory with algorithmic information theory.