Bayesian Estimation, Conjugacy & Loss Function Decision Theory

In the Bayesian framework, all uncertain quantities—both observable experimental data $\mathbf{X}$ and unobservable parameters $\boldsymbol{\theta}$—are embedded within a unified joint probability space $(\mathcal{X} \times \Theta, \mathcal{F}, \mathbb{P})$.

The Components of a Bayesian Model:

1. The Prior Distribution: $\pi(\boldsymbol{\theta})$ denotes the probability density function (or probability measure) defined on the parameter space $\Theta$. It formalizes all knowledge, physical domain constraints, historical experimental data, or subjective uncertainty regarding $\boldsymbol{\theta}$ prior to observing the current sample $\mathbf{X}$.

2. The Sampling Model (Likelihood): $f(\mathbf{x} \mid \boldsymbol{\theta})$ represents the conditional probability density of the data $\mathbf{X} = \mathbf{x}$ given the parameter $\boldsymbol{\theta}$. When viewed as a function of $\boldsymbol{\theta}$ for fixed observed data $\mathbf{x}$, it is the likelihood function $L(\boldsymbol{\theta}; \mathbf{x})$.

3. The Marginal Likelihood (Evidence / Normalizing Constant): The total integrated probability of the observed data across all candidate states of nature is:

In discrete models, the integral is replaced by countable summation. The evidence $m(\mathbf{x})$ acts as a normalizer ensuring that the posterior integrates to unity.

4. The Posterior Distribution: Combining prior beliefs with empirical evidence via Bayes' Theorem produces the conditional distribution of $\boldsymbol{\theta}$ given the observed data $\mathbf{X} = \mathbf{x}$:

The Proportionality Principle

Because the marginal evidence $m(\mathbf{x})$ is a deterministic constant with respect to $\boldsymbol{\theta}$ once the data $\mathbf{x}$ are observed, Bayesian inference is almost universally summarized by the fundamental proportionality:

The posterior distribution $\pi(\boldsymbol{\theta} \mid \mathbf{x})$ constitutes the complete, definitive answer to any Bayesian estimation problem. It encapsulates all available information regarding $\boldsymbol{\theta}$ in light of both prior knowledge and empirical observations. Any point estimate, interval estimate, or decision is derived as an optimal summary of this posterior density.

The choice of prior distribution $\pi(\boldsymbol{\theta})$ is central to Bayesian methodology. Depending on the scientific context, priors are classified into three major categories:

1. Informative Priors

When prior clinical trials, historical telemetry, or physical laws provide tangible quantitative information, experts calibrate hyperparameters (e.g., prior mean $\mu_0$ and prior precision $\tau_0$) to match empirical moments. Informative priors allow historical data to directly improve estimation precision in small current samples.

2. Vague / Diffuse Priors

When an investigator has minimal advance knowledge, they may assign a prior with immense dispersion (such as a Gaussian prior $\mathcal{N}(0, 10^6)$ or a Uniform prior on a wide interval $[-M, M]$). In such cases, the posterior distribution is dominated entirely by the likelihood function.

3. Objective / Non-Informative Priors (Jeffreys' Invariant Rule)

A naive approach to non-informative priors is assigning a flat, uniform prior $\pi(\theta) \propto 1$ across the parameter space (Laplace's Principle of Insufficient Reason). However, this rule suffers from a severe mathematical contradiction: a uniform prior is not transformation-invariant! If $\pi_\theta(\theta) = 1$ is flat, and we reparameterize the problem by a non-linear mapping $\psi = g(\theta)$ (e.g., $\psi = \theta^2$ or $\psi = 1/\theta$), the Jacobian rule of density transformation states:

The prior is non-uniform in $\psi$! Claiming total ignorance about $\theta$ mathematically forces the investigator to assert strong prior opinions about $\psi = \theta^2$.

Theorem: Jeffreys' Invariant Prior Rule (1946)

Sir Harold Jeffreys resolved this paradox by defining an objective prior directly proportional to the square root of the Fisher Information:

where $\mathbf{I}(\boldsymbol{\theta})$ is the Fisher Information matrix per observation.

Proof of Invariance:

Let $\boldsymbol{\psi} = \mathbf{g}(\boldsymbol{\theta})$ be a smooth, one-to-one reparameterization with Jacobian $\mathbf{J} = \frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\psi}}$. From Section 2.7, Fisher Information transforms tensorially as:

Taking determinants on both sides:

Taking the square root:

This matches the exact multivariable change-of-variables formula for probability densities! Jeffreys' prior is intrinsically invariant under all smooth reparameterizations of the parameter space. Classical Jeffreys' priors include:

• Location parameter $\mu$ in $\mathcal{N}(\mu, \sigma^2)$: $I(\mu) = 1/\sigma^2$ (constant), so $\pi_J(\mu) \propto 1$ (flat Lebesgue measure on $\mathbb{R}$).
• Scale parameter $\sigma$ in $\mathcal{N}(\mu, \sigma^2)$: $I(\sigma) = 2/\sigma^2$, so $\pi_J(\sigma) \propto 1/\sigma$ (log-uniform prior on $(0, \infty)$).
• Binomial success probability $p$: $I(p) = \frac{1}{p(1-p)}$, so $\pi_J(p) \propto \frac{1}{\sqrt{p(1-p)}} = \text{Beta}(1/2, 1/2)$ (arcsine prior).

Bernardo's Objective Reference Priors in Multiparameter Models

While Jeffreys' prior $\pi_J(\boldsymbol{\theta}) \propto \sqrt{\det(\mathbf{I}(\boldsymbol{\theta}))}$ works exquisitely in single-parameter problems, it can fail catastrophically in multiparameter settings. In a famous counterexample, Charles Stein considered the normal model:

where the parameter of interest is the squared length $\eta = \sum_{i=1}^p \mu_i^2$. Evaluating Jeffreys' joint prior yields $\pi_J(\mu_1, \dots, \mu_p, \sigma) \propto \sigma^{-(p+1)}$. When $p$ is large (e.g., $p = 50$), the posterior distribution for $\eta$ suffers from severe marginalization paradoxes, producing posterior credible intervals that are systematically shifted away from the true parameter value. To overcome this defect, José-Miguel Bernardo (1979), with further developments by James Berger, formulated the theory of Reference Priors. Instead of treating all parameters symmetrically, reference analysis partitions the parameters into ordered groups:

where $\theta_1$ is the primary parameter of interest and $\theta_2, \dots, \theta_k$ are nuisance parameters. The reference prior is derived by maximizing the expected Kullback–Leibler divergence (Shannon mutual information) between the prior and the posterior distribution:

subject to a sequence of compact nested subsets. By maximizing information flow sequentially from data to parameter, Bernardo's reference algorithm eliminates marginalization paradoxes and produces objective Bayesian credible intervals that achieve exact frequentist coverage to higher orders of asymptotic approximation.

When evaluating the posterior distribution analytically, integrating complex likelihoods can yield intractable normalizing constants. In 1961, Howard Raiffa and Robert Schlaifer introduced the concept of conjugate prior distributions.

Definition of Conjugacy

A family of prior probability distributions $\mathcal{P}_{\text{prior}}$ on $\Theta$ is said to be conjugate to a parametric sampling model $\mathcal{F} = \{f(\mathbf{x} \mid \boldsymbol{\theta})\}$ if for every prior $\pi(\boldsymbol{\theta}) \in \mathcal{P}_{\text{prior}}$ and every possible data realization $\mathbf{x}$, the resulting posterior distribution $\pi(\boldsymbol{\theta} \mid \mathbf{x})$ belongs to the very same distribution family $\mathcal{P}_{\text{prior}}$.

Under conjugacy, updating prior beliefs in light of data reduces to an algebraic update of hyperparameters:

Conjugacy in Natural Exponential Families

A foundational theorem established by George A. Barnard and refined by Diaconis and Ylvisaker (1979) proves that every natural exponential family admits a natural conjugate prior. Consider a single-parameter exponential family in canonical form:

For an i.i.d. sample of size $n$, the joint likelihood is:

Define the natural conjugate prior parameterized by hyperparameters $(n_0, t_0)$:

Multiply prior and likelihood to obtain the posterior:

Notice the analytical form: this is identical to the prior family with updated hyperparameters:

The hyperparameter $n_0$ represents the effective prior sample size, and $t_0$ represents the effective prior sum of observations. Bayesian updating simply sums the pseudo-observations from the prior and the real observations from the sample!

Canonical Conjugate Pairs:

1. Bernoulli / Binomial Likelihood $\leftrightarrow$ Beta Prior:

2. Poisson Likelihood $\leftrightarrow$ Gamma Prior:

3. Normal Likelihood (Mean unknown, variance $\sigma_0^2$ known) $\leftrightarrow$ Normal Prior:

4. Exponential / Gamma Likelihood (Scale unknown) $\leftrightarrow$ Inverse-Gamma Prior:

How does a Bayesian extract an optimal single numerical point estimate $\hat{\theta}$ from the full posterior distribution $\pi(\theta \mid \mathbf{x})$? In Bayesian decision theory, point estimation is formalized as a statistical game against nature.

The Decision-Theoretic Framework

Let $\mathcal{A} \subseteq \mathbb{R}$ denote the action space (the set of possible point estimates $a = \hat{\theta}$). Let $L(\theta, a)$ be a designated loss function measuring the economic or scientific penalty incurred by choosing estimate $a$ when the true state of nature is $\theta$. The Posterior Expected Loss (or posterior risk) of an action $a$ given observed data $\mathbf{x}$ is:

The Bayes Point Estimator $\hat{\theta}_{\text{Bayes}}(\mathbf{x})$ is the decision rule that minimizes the posterior expected loss for the realized sample $\mathbf{x}$:

Integrated Bayes Risk

Prior to data collection, the overall expected risk of an estimation rule $\delta(\mathbf{X})$ across all conceivable data realizations and parameter values is the Bayes Risk:

By Fubini's Theorem, interchanging the order of integration reveals that minimizing $r(\pi, \delta)$ globally across all decision rules $\delta$ is mathematically achieved by minimizing the posterior expected loss $\rho(a, \mathbf{x})$ pointwise for each individual observed dataset $\mathbf{x}$.

The choice of loss function $L(\theta, a)$ dictates the optimal summary of the posterior distribution:

1. Squared Error Loss $\implies$ Posterior Mean

Let $L(\theta, a) = (a - \theta)^2$. The posterior expected loss is:

Differentiating with respect to $a$ and setting to zero:

Since $\int \pi(\theta \mid \mathbf{x}) \, d\theta = 1$:

The second derivative is $\frac{d^2\rho}{da^2} = 2 > 0$, confirming a unique global minimum. Theorem: Under Squared Error Loss, the optimal Bayes point estimator is uniquely the Posterior Mean:

2. Absolute Error Loss $\implies$ Posterior Median

Let $L(\theta, a) = |a - \theta|$. The posterior expected loss is:

Differentiating with respect to $a$ via Leibniz's rule:

Setting the derivative to zero:

Theorem: Under Absolute Error Loss, the optimal Bayes point estimator is the Posterior Median:

3. Zero-One Loss $\implies$ Maximum A Posteriori (MAP) Mode

Let $L_\epsilon(\theta, a) = \begin{cases} 0 & \text{if } |a - \theta| \le \epsilon \\ 1 & \text{if } |a - \theta| > \epsilon \end{cases}$. Taking the limit as the tolerance width $\epsilon \to 0^+$: Minimizing posterior loss is equivalent to maximizing the posterior probability mass in an infinitesimal neighborhood around $a$. Theorem: Under Zero-One Loss, the optimal Bayes estimator is the Maximum A Posteriori (MAP) Estimator (the mode of the posterior):

Notice that if the prior is flat ($\pi(\theta) \propto 1$), the MAP estimator coincides identically with the Maximum Likelihood Estimator ($\hat{\theta}_{\text{MAP}} = \hat{\theta}_{\text{MLE}}$)!

Asymmetric Loss Functions: The LINEX Loss Model

In engineering risk analysis and financial portfolio management, the consequences of overestimation and underestimation are rarely symmetric. For example:

• In dam design, underestimating the 100-year flood level leads to catastrophic structural overtopping and loss of life. Overestimating the flood level merely leads to modest financial expenditure in building a higher concrete wall.
• In pharmaceutical manufacturing, underestimating toxic contaminant concentrations is fatal; overestimating causes harmless batch scrapping.

To accommodate directional penalties, Varian (1975) introduced the Linear-Exponential (LINEX) Loss Function:

where $\Delta = \hat{\theta} - \theta$ is the estimation error. Notice the behavior of the LINEX loss:

• If $a > 0$, an overestimation error ($\Delta > 0$) causes exponential penalty growth $e^{a \Delta}$, while an underestimation error ($\Delta < 0$) causes only linear penalty growth $-a \Delta$.
• If $a < 0$, underestimation is penalized exponentially.
• As $a \to 0$, Taylor expanding $e^{a \Delta} = 1 + a \Delta + \frac{a^2 \Delta^2}{2} + \mathcal{O}(a^3)$ shows that $L(\Delta) \approx \frac{b a^2}{2} \Delta^2$, which reproduces standard squared error loss.

Theorem: The Optimal Bayes Estimator under LINEX Loss

The posterior expected loss is:

Differentiating with respect to $a$ and setting to zero:

Taking the natural logarithm on both sides:

Solving for the optimal point estimate $\hat{\theta}_{\text{LINEX}}$:

Notice the extraordinary analytical beauty of this result: $\mathbb{E}[e^{-a \theta} \mid \mathbf{x}] = M_{\theta \mid \mathbf{x}}(-a)$ is precisely the Moment Generating Function of the posterior distribution evaluated at $-a$! If the posterior distribution is Gaussian $\theta \mid \mathbf{x} \sim \mathcal{N}(\mu_n, \sigma_n^2)$, its MGF is $M(-a) = \exp\left( -a \mu_n + \frac{a^2 \sigma_n^2}{2} \right)$. Substituting this into the LINEX formula:

The optimal estimate is the posterior mean $\mu_n$ shifted by an exact risk premium $\frac{a}{2} \sigma_n^2$ proportional to the posterior variance! If underestimation is dangerous ($a < 0$), the estimator automatically shifts upward by $\frac{|a|}{2} \sigma_n^2$ to safeguard against failure.

In Bayesian statistics, parameter uncertainty after observing data is directly communicated through Credible Intervals.

Definition of a Credible Region

A subset $C \subseteq \Theta$ is called a $100(1 - \alpha)\%$ Bayesian Credible Region for $\boldsymbol{\theta}$ if its posterior probability equals $1 - \alpha$:

Epistemological Contrast with Frequentist Intervals:

• Frequentist Interval $[L(\mathbf{x}), U(\mathbf{x})]$: The endpoints are random; the parameter is fixed. The probability statement $\mathbb{P}(L \le \theta \le U) = 0.95$ applies to the long-run coverage rate of the random procedure over infinite hypothetical repetitions. For a realized numerical interval $[12.4, 15.8]$, the parameter is either inside or outside.
• Bayesian Credible Interval: The data $\mathbf{x}$ are fixed; the parameter $\theta$ is random. The probability statement:

is literally and directly true! The investigator can state that given the realized data and prior, the subjective probability that $\theta$ lies in $[12.4, 15.8]$ is exactly $95\%$.

Highest Posterior Density (HPD) Regions

Among all possible subsets satisfying $\mathbb{P}(\theta \in C \mid \mathbf{x}) = 1 - \alpha$, which one should be chosen? The optimal choice is the Highest Posterior Density (HPD) Region, which minimizes the total volume (length) of the credible set:

where $k_\alpha$ is the horizontal density cutoff chosen such that $\int_{C_{\text{HPD}}} \pi(\boldsymbol{\theta} \mid \mathbf{x}) \, d\boldsymbol{\theta} = 1 - \alpha$. HPD regions possess two defining geometric properties:

1. Every parameter value inside the region has a higher posterior density than any point outside the region.
2. The HPD region achieves the shortest possible expected volume among all credible sets of the same nominal level $1 - \alpha$.

If the posterior distribution is symmetric and unimodal (such as Gaussian or Student's $t$), the HPD region coincides with the equal-tailed credible interval. If the posterior is skewed or bimodal, the HPD region naturally splits into disjoint intervals, reflecting multiple plausible parameter clusters.

What happens to Bayesian estimators as the sample size $n$ approaches infinity? Do prior beliefs permanently bias the scientific conclusion?

The celebrated Bernstein–von Mises Theorem (Laplace, 1810; Bernstein, 1917; von Mises, 1931) proves that as $n \to \infty$, the posterior distribution forgets the prior entirely and converges to a normal distribution centered at the MLE!

Theorem: The Bernstein–von Mises Theorem

Let $X_1, \dots, X_n \stackrel{\text{i.i.d.}}{\sim} P_{\theta_0}$ satisfy standard Cramér regularity conditions. Let the prior density $\pi(\theta)$ be continuous and strictly positive in an open neighborhood of the true parameter $\theta_0$ ($\pi(\theta_0) > 0$). Let $\hat{\theta}_n$ be the Maximum Likelihood Estimator, and let $I_1(\theta_0)$ be the Fisher Information per observation. Then the total variation distance between the posterior distribution of the localized parameter $h = \sqrt{n}(\theta - \hat{\theta}_n)$ and the standard Gaussian distribution converges to zero in probability:

Profound Implications:

1. Prior Disappearance: The influence of any reasonable prior $\pi(\theta)$ decays as $\mathcal{O}(1/n)$. In large samples, the prior vanishes, and data dominate the posterior completely.

2. Asymptotic Equivalence of Frequentist and Bayesian Inference: In large samples:

• The Bayes posterior mean $\mathbb{E}[\theta \mid \mathbf{X}]$ converges to the MLE $\hat{\theta}$.
• The posterior variance converges to the inverse Fisher Information $\frac{1}{n I_1(\theta_0)}$.
• The Bayesian $95\%$ HPD credible interval converges identically to the frequentist $95\%$ Wald confidence interval!

This grand synthesis proves that frequentist and Bayesian methodologies reach identical inferential conclusions in large samples.

In classical Bayesian analysis, hyperparameters are fixed a priori. In Empirical Bayes (EB), introduced by Herbert Robbins (1956), hyperparameters are estimated directly from the ensemble of observed data across multiple experimental units, enabling borrowing of strength.

The James–Stein Paradox (1961)

Consider observing $p$ independent physical quantities:

We wish to estimate the vector $\boldsymbol{\theta} = (\theta_1, \dots, \theta_p)^T$ under total squared error loss:

The natural point estimator is the Maximum Likelihood Estimator (and UMVUE):

Its expected risk is constant across all $\boldsymbol{\theta} \in \mathbb{R}^p$:

For over forty years, mathematical statisticians assumed that $\mathbf{X}$ was the undisputed optimal estimator. In 1961, Charles Stein and Willard James proved a shocking mathematical theorem that shook the foundations of mathematical statistics: For dimension $p \ge 3$, the Maximum Likelihood Estimator $\mathbf{X}$ is strictly inadmissible!

The James–Stein Shrinkage Estimator

James and Stein constructed the non-linear shrinkage estimator:

Stein proved that for all $p \ge 3$ and for every single parameter vector $\boldsymbol{\theta} \in \mathbb{R}^p$:

The James–Stein estimator achieves strictly lower total mean squared error everywhere in $\mathbb{R}^p$. The Bayesian derivation of James–Stein reveals that it is precisely an Empirical Bayes estimator: placing an exchangeable prior $\theta_i \sim \mathcal{N}(0, \tau^2)$ on the parameters and estimating the prior variance $\tau^2$ from the marginal norm $\|\mathbf{X}\|^2$ naturally produces the shrinkage factor $1 - \frac{p - 2}{\|\mathbf{X}\|^2}$. Shrinking raw estimates toward a common ensemble mean eliminates high-dimensional noise.

When hierarchical or non-conjugate priors are specified, the marginal evidence $m(\mathbf{x}) = \int f(\mathbf{x} \mid \boldsymbol{\theta}) \pi(\boldsymbol{\theta}) \, d\boldsymbol{\theta}$ involves an intractable high-dimensional integral. In the 1990s, the adoption of Markov Chain Monte Carlo (MCMC) algorithms revolutionized applied statistics by allowing researchers to sample directly from the posterior distribution $\pi(\boldsymbol{\theta} \mid \mathbf{x})$ without ever calculating the normalizing constant $m(\mathbf{x})$.

1. The Metropolis–Hastings Algorithm (Metropolis et al., 1953; Hastings, 1970)

To sample from an intractable target density $\pi(\boldsymbol{\theta} \mid \mathbf{x}) = \frac{p(\boldsymbol{\theta}, \mathbf{x})}{m(\mathbf{x})}$: At iteration $t$ with current state $\boldsymbol{\theta}^{(t)}$:

1. Draw a candidate proposal $\boldsymbol{\theta}^*$ from a proposal transition distribution $q(\boldsymbol{\theta}^* \mid \boldsymbol{\theta}^{(t)})$.
2. Compute the Metropolis–Hastings Acceptance Ratio:

Notice that the intractable normalizer $m(\mathbf{x})$ cancels out completely!

1. Draw $U \sim \text{Uniform}(0, 1)$. If $U \le \alpha$, accept the proposal and set $\boldsymbol{\theta}^{(t+1)} = \boldsymbol{\theta}^*$; otherwise reject and set $\boldsymbol{\theta}^{(t+1)} = \boldsymbol{\theta}^{(t)}$.

By the detailed balance condition, the stationary distribution of the generated ergodic Markov chain $\{\boldsymbol{\theta}^{(t)}\}$ is uniquely the true posterior distribution $\pi(\boldsymbol{\theta} \mid \mathbf{x})$.

2. The Gibbs Sampler (Geman and Geman, 1984; Gelfand and Smith, 1990)

When the parameter vector $\boldsymbol{\theta} = (\theta_1, \dots, \theta_k)^T$ can be decomposed such that the full conditional distributions $\pi(\theta_j \mid \boldsymbol{\theta}_{-j}, \mathbf{x})$ are known standard distributions, the Gibbs sampler updates parameters cyclically one variable at a time:

• Draw $\theta_1^{(t+1)} \sim \pi(\theta_1 \mid \theta_2^{(t)}, \theta_3^{(t)}, \dots, \theta_k^{(t)}, \mathbf{x})$
• Draw $\theta_2^{(t+1)} \sim \pi(\theta_2 \mid \theta_1^{(t+1)}, \theta_3^{(t)}, \dots, \theta_k^{(t)}, \mathbf{x})$
• $\dots$
• Draw $\theta_k^{(t+1)} \sim \pi(\theta_k \mid \theta_1^{(t+1)}, \theta_2^{(t+1)}, \dots, \theta_{k-1}^{(t+1)}, \mathbf{x})$

Because proposals in Gibbs sampling are drawn directly from exact full conditionals, the acceptance probability is identically $1$ (zero rejected draws).

MCMC Convergence Diagnostics: Gelman-Rubin $\hat{R}$ and Effective Sample Size

When executing MCMC sampling (such as Metropolis–Hastings or Gibbs sampling), the generated Markov chain requires an initial burn-in phase to reach its stationary distribution. Furthermore, successive MCMC draws are autocorrelated rather than independent. Statisticians deploy two essential convergence diagnostics:

1. The Gelman–Rubin Potential Scale Reduction Factor ($\hat{R}$)

Andrew Gelman and Donald Rubin (1992) proposed running $M \ge 3$ independent parallel chains starting from dispersed initial values. Let $N$ denote the number of iterations per chain after discarding burn-in.

• Calculate the between-chain variance $B$:

• Calculate the average within-chain variance $W$:

• The pooled marginal posterior variance estimate is:

The potential scale reduction factor is:

If the chains have mixed and converged to the target distribution, between-chain variance matches within-chain variance, so $\hat{R} \approx 1.0$. In modern Bayesian practice, convergence is diagnosed only when $\hat{R} < 1.05$ across all monitored parameters.

2. Effective Sample Size (ESS)

Due to positive autocorrelation $\rho_k$ between successive iterations, $N$ dependent MCMC draws contain less information than $N$ independent draws. The integrated autocorrelation time is:

The Effective Sample Size (ESS) is:

The ESS quantifies the equivalent number of independent Monte Carlo samples generated by the chain. For reliable posterior credible intervals, practitioners demand $\text{ESS} \ge 1,000$.

We conclude the mathematical foundations of statistical estimation by connecting Bayesian decision theory back to frequentist admissibility.

Definition of Admissibility

Let $R(\theta, \delta) = \mathbb{E}_\theta[L(\theta, \delta(\mathbf{X}))]$ denote the frequentist risk function of an estimator $\delta$. An estimator $\delta$ is said to be inadmissible if there exists another estimator $\delta^*$ such that:

with strict inequality $R(\theta_0, \delta^*) < R(\theta_0, \delta)$ for at least one parameter value $\theta_0$. If no such dominating estimator $\delta^*$ exists, $\delta$ is admissible.

Theorem: Wald's Complete Class Theorem (1950)

In any statistical decision problem with a compact parameter space and continuous risk function:

1. Every Bayes estimator with respect to a proper prior with full support is admissible.
2. Every admissible estimator is either a Bayes estimator or the limit of a sequence of Bayes estimators.

Wald's Complete Class Theorem establishes the absolute mathematical supremacy of Bayesian decision rules: If an analyst wishes to construct an optimal, non-dominated estimator in any scientific or engineering problem, that estimator must be derived as a Bayes estimator under some prior distribution! Any estimator that cannot be represented as a Bayes rule is mathematically defective and can be improved upon.