Architecture of Statistical Hypothesis Testing: Neyman-Pearson Paradigm, Power Functions & P-Values

In mathematical statistics, inferential theory is divided into two complementary branches: parameter estimation, where observed data are used to construct numerical approximations or plausible sets for unknown parameters, and statistical hypothesis testing, where empirical observations are evaluated to adjudicate between competing structural assertions about the underlying probability law governing an experiment. While estimation addresses the question of what value a parameter assumes, hypothesis testing addresses a binary decision problem regarding whether the true state of nature resides within a specified subset of the parameter manifold.

Let $(\mathcal{X}, \mathcal{A}, \mathcal{P})$ denote a parametric statistical model, where $\mathcal{X} \subseteq \mathbb{R}^n$ is the sample space of an observable random vector $\mathbf{X} = (X_1, X_2, \dots, X_n)^T$, $\mathcal{A}$ is a $\sigma$-algebra of measurable events on $\mathcal{X}$, and $\mathcal{P} = \{P_\theta : \theta \in \Theta\}$ is a family of candidate probability distributions indexed by a parameter vector $\theta$ belonging to the parameter space $\Theta \subseteq \mathbb{R}^k$. A statistical hypothesis is any formal, falsifiable assertion concerning the probability distribution $P_\theta$ of the observable random vector $\mathbf{X}$. In a parametric setting, because the probability law $P_\theta$ is completely indexed by $\theta$, every statistical hypothesis corresponds bijectively to an assertion that the true parameter lies within a particular nonempty subset of $\Theta$.

Formally, the parameter space $\Theta$ is partitioned into two mutually exclusive and exhaustive subsets, denoted by $\Theta_0$ and $\Theta_1$, satisfying the set-theoretic conditions:

The competing assertions are then formulated as the null hypothesis, denoted $H_0$, and the alternative hypothesis (or research hypothesis), denoted $H_1$ (or $H_a$):

The structural complexity of a hypothesis is governed by the cardinality and topological dimension of its associated parameter subset:

1. Simple Hypothesis: A statistical hypothesis is termed simple if it completely specifies the probability distribution of the observable sample $\mathbf{X}$ without leaving any unknown nuisance parameters or free coordinates. Equivalently, $H_0 : \theta \in \Theta_0$ is simple if and only if $\Theta_0 = \{\theta_0\}$ is a singleton set containing a single fully specified point in $\Theta$. For example, in an independent and identically distributed Gaussian model $X_1, \dots, X_n \sim \mathcal{N}(\mu, \sigma^2)$ where the variance $\sigma^2 = \sigma_0^2$ is known, the assertion $H_0 : \mu = 0$ is a simple hypothesis because the joint density on $\mathbb{R}^n$ is uniquely determined.

2. Composite Hypothesis: A statistical hypothesis is termed composite if the corresponding parameter subset contains more than one element, so that the exact probability distribution of the sample remains incompletely specified even when the hypothesis holds true. Composite hypotheses arise in two principal ways: either the parameter of interest ranges over an interval—such as a one-sided alternative $H_1 : \theta > \theta_0$ or a two-sided alternative $H_1 : \theta \neq \theta_0$—or the model contains unspecified nuisance parameters. For instance, in a normal population $\mathcal{N}(\mu, \sigma^2)$ where both $\mu \in \mathbb{R}$ and $\sigma^2 > 0$ are unknown, the hypothesis $H_0 : \mu = 0$ is composite because the null subspace $\Theta_0 = \{(0, \sigma^2) : \sigma^2 > 0\}$ is a one-dimensional half-line rather than a single point.

Non-Randomized Tests and Sample Space Partitions

In its classical deterministic formulation, a statistical hypothesis test is a measurable rule that partitions the sample space $\mathcal{X}$ into two disjoint Borel sets: the acceptance region (or non-rejection region) $\mathcal{C}^c \in \mathcal{A}$ and the critical region (or rejection region) $\mathcal{C} \in \mathcal{A}$, satisfying $\mathcal{X} = \mathcal{C} \cup \mathcal{C}^c$ and $\mathcal{C} \cap \mathcal{C}^c = \emptyset$. When the experiment yields a sample realization $\mathbf{x} \in \mathcal{X}$, the investigator applies the binary decision rule:

• If $\mathbf{x} \in \mathcal{C}$, the null hypothesis $H_0$ is rejected in favor of the alternative $H_1$.
• If $\mathbf{x} \in \mathcal{C}^c$, the investigator fails to reject the null hypothesis $H_0$.

In practice, the critical region $\mathcal{C}$ is almost always defined through a scalar or vector-valued test statistic $T : \mathcal{X} \to \mathbb{R}^m$, which compresses the $n$-dimensional sample vector $\mathbf{X}$ into a summary measure of departure from the null hypothesis, such that $\mathcal{C} = \{\mathbf{x} \in \mathcal{X} : T(\mathbf{x}) \in R\}$ for a rejection region $R \subseteq \mathbb{R}^m$.

In scientific epistemology and Popperian falsificationism, the expressions "accepting $H_0$" and "failing to reject $H_0$" carry distinct interpretations. While Abraham Wald's decision-theoretic framework treats hypothesis testing as a two-action decision space $\mathcal{D} = \{d_0, d_1\}$ where $d_0$ denotes acting as though $H_0$ is true and $d_1$ denotes acting as though $H_1$ is true, empirical data of finite sample size can rarely prove the exact validity of a point null hypothesis $\theta = \theta_0$ against arbitrary infinitesimal perturbations $\theta = \theta_0 + \varepsilon$. Consequently, falling inside $\mathcal{C}^c$ signifies that the empirical evidence is insufficient to refute $H_0$ beyond the prescribed threshold of reasonable doubt, rather than establishing that $H_0$ is literally true with certainty.

To construct a unified mathematical theory of hypothesis testing that encompasses both continuous and discrete probability distributions and permits the application of functional analysis and convex optimization, Jerzy Neyman and Egon Pearson replaced the rigid indicator set $\mathcal{C}$ with the more general concept of a **randomized test function**. A **test function** (or critical function) is a Borel-measurable mapping $\phi : \mathcal{X} \to [0, 1]$ that assigns to every possible sample realization $\mathbf{x} \in \mathcal{X}$ a conditional probability $\phi(\mathbf{x})$ of rejecting the null hypothesis $H_0$ given that $\mathbf{X} = \mathbf{x}$ has been observed. Once the data vector $\mathbf{x}$ is recorded, the investigator performs an auxiliary independent Bernoulli trial—generating a uniform random variable $U \sim \text{Uniform}(0, 1)$—and rejects $H_0$ if $U \le \phi(\mathbf{x})$, while failing to reject $H_0$ if $U > \phi(\mathbf{x})$. - A **non-randomized test** is the special case in which the test function takes values exclusively in the binary set $\{0, 1\}$. In that setting, $\phi(\mathbf{x}) = \mathbf{1}_{\mathcal{C}}(\mathbf{x})$ is simply the indicator function of the critical region $\mathcal{C} = \{\mathbf{x} \in \mathcal{X} : \phi(\mathbf{x}) = 1\}$. - A **randomized test** takes fractional values $0 < \phi(\mathbf{x}) < 1$ on a boundary subset of the sample space (typically where a discrete test statistic equals a critical threshold). The space of all measurable test functions $\Phi = \{\phi : \mathcal{X} \to [0, 1]\}$ forms a convex, weak-star compact subset of $L^\infty(\mathcal{X}, \mathcal{A}, \mu)$, which guarantees the mathematical existence of optimal tests across arbitrary probability spaces. ### Type I and Type II Errors and Asymmetric Loss Structure Because the sample $\mathbf{X}$ exhibits random variability, any decision rule $\phi$ is susceptible to two fundamentally distinct modes of inferential error depending on the true, unobserved location of the parameter $\theta$: 1. **Type I Error (False Positive / False Discovery)**: Rejecting the null hypothesis $H_0$ when the true parameter actually resides in the null subspace ($\theta \in \Theta_0$). For a given parameter point $\theta \in \Theta_0$, the probability of committing a Type I error is the expected value of the test function under $P_\theta$: $$\alpha(\theta) = \mathbb{P}_\theta(\text{Reject } H_0) = \mathbb{E}_\theta[\phi(\mathbf{X})] = \int_{\mathcal{X}} \phi(\mathbf{x}) \, dP_\theta(\mathbf{x}), \quad \theta \in \Theta_0$$ 2. **Type II Error (False Negative / Missed Detection)**: Failing to reject the null hypothesis $H_0$ when the true parameter actually resides in the alternative subspace ($\theta \in \Theta_1$). For a given parameter point $\theta \in \Theta_1$, the probability of committing a Type II error is: $$\tilde{\beta}(\theta) = \mathbb{P}_\theta(\text{Fail to Reject } H_0) = \mathbb{E}_\theta[1 - \phi(\mathbf{X})] = 1 - \mathbb{E}_\theta[\phi(\mathbf{X})], \quad \theta \in \Theta_1$$ In a decision-theoretic framework with loss function $L(\theta, d)$, where $L(\theta, d_0) = 0$ for $\theta \in \Theta_0$ and $L(\theta, d_1) = c_{\text{I}} > 0$ for $\theta \in \Theta_0$, while $L(\theta, d_1) = 0$ for $\theta \in \Theta_1$ and $L(\theta, d_0) = c_{\text{II}} > 0$ for $\theta \in \Theta_1$, the risk function $R(\theta, \phi) = \mathbb{E}_\theta[L(\theta, \phi(\mathbf{X}))]$ reduces directly to $c_{\text{I}} \alpha(\theta)$ on $\Theta_0$ and $c_{\text{II}} \tilde{\beta}(\theta)$ on $\Theta_1$. Ideally, an investigator would seek a test function $\phi$ that simultaneously drives both $\alpha(\theta)$ and $\tilde{\beta}(\theta)$ to zero for a fixed sample size $n$. However, for any non-trivial statistical model where the measures $P_{\theta_0}$ and $P_{\theta_1}$ are mutually absolutely continuous (or share overlapping support), these two error probabilities stand in strict mathematical opposition. Shrinking the critical region $\mathcal{C}$ to reduce the probability of a Type I error inevitably enlarges the acceptance region $\mathcal{C}^c$, thereby inflating the probability of a Type II error. Conversely, expanding $\mathcal{C}$ to capture more alternative realizations increases the false positive rate under $H_0$. Only by increasing the total Fisher information through a larger sample size $n$ can both error rates be reduced simultaneously. To resolve this fundamental trade-off for a fixed sample size $n$, the **Neyman-Pearson paradigm** introduces an asymmetric optimization hierarchy: - The null hypothesis $H_0$ is designated as the conservative baseline status quo (analogous to the presumption of innocence in jurisprudence or the absence of therapeutic efficacy in a clinical trial) whose false rejection carries severe scientific or regulatory consequences. - We first impose a strict, predetermined upper bound $\alpha \in (0, 1)$—called the **level of significance**—on the permissible probability of a Type I error across the entire null parameter space: $$\sup_{\theta \in \Theta_0} \mathbb{E}_\theta[\phi(\mathbf{X})] \le \alpha$$ - Subject to this hard constraint on Type I error, we then search within the feasible class of level-$\alpha$ tests for a rule $\phi^*$ that minimizes the Type II error probability $\tilde{\beta}(\theta)$ (or equivalently maximizes detection probability) across $\theta \in \Theta_1$.

The complete frequentist performance of any statistical test $\phi : \mathcal{X} \to [0, 1]$ across the entire parameter space $\Theta = \Theta_0 \cup \Theta_1$ is encapsulated by a single analytical mapping known as the power function.

Definition of the Power Function and Operating Characteristic Curve

For any test function $\phi(\mathbf{X})$, the power function $\beta_\phi : \Theta \to [0, 1]$ (often written simply as $\beta(\theta)$ when $\phi$ is clear from context) is defined for every $\theta \in \Theta$ as the expected value of $\phi(\mathbf{X})$ under the probability measure $P_\theta$:

When $\phi(\mathbf{x}) = \mathbf{1}_{\mathcal{C}}(\mathbf{x})$ is a non-randomized test with critical region $\mathcal{C}$, the power function reduces to the exact probability that the sample falls inside the rejection region when $\theta$ is the true parameter:

Note carefully the dual role played by $\beta(\theta)$ across the two partitions of the parameter space:

• When $\theta \in \Theta_0$, $\beta(\theta)$ equals the Type I error probability at $\theta$.
• When $\theta \in \Theta_1$, $\beta(\theta)$ equals the power of the test against the specific alternative $\theta$, which is $1 - \tilde{\beta}(\theta)$, the probability of correctly detecting a false null hypothesis.

In industrial quality control, reliability engineering, and sequential analysis, analysts frequently examine the complement of the power function, known as the Operating Characteristic (OC) curve, denoted $L(\theta)$:

For an ideal test with infinite information, the power function would coincide with the indicator function of the alternative space, $\beta_{\text{ideal}}(\theta) = \mathbf{1}_{\Theta_1}(\theta)$, vanishing identically on $\Theta_0$ and jumping discontinuously to unity on $\Theta_1$. For finite sample sizes in regular parametric families where $f(\mathbf{x}; \theta)$ varies smoothly with $\theta$, dominated convergence implies that $\beta(\theta)$ is a continuous—and frequently real-analytic—function of $\theta$. Because a continuous function cannot jump instantaneously from $0$ to $1$ at the boundary separating $\Theta_0$ and $\Theta_1$, any finite-sample test has power close to $\alpha$ for alternative parameter values $\theta \in \Theta_1$ located in the immediate topological neighborhood of $\Theta_0$.

Exact Distinction Between Test Size and Significance Level

A crucial theoretical distinction separates the size of a test from its level of significance:

1. Level of Significance ($\alpha$): A test $\phi$ is said to have significance level $\alpha \in [0, 1]$ (or to be a level-$\alpha$ test) if its power function never exceeds $\alpha$ anywhere in the null parameter space:

2. Size of a Test ($\alpha_{\text{size}}$): The size of a test $\phi$ is defined as the supremum of the Type I error probability over the entire null parameter space $\Theta_0$:

Every size-$\alpha$ test is automatically a level-$\alpha$ test, and any test of size $\alpha_0 \le \alpha$ is a valid level-$\alpha$ test. However, a level-$\alpha$ test may have a strict inequality $\sup_{\theta \in \Theta_0} \beta(\theta) < \alpha$. This distinction arises constantly in two settings:

• Discrete Non-Randomized Tests: If $X \sim \text{Binomial}(n = 10, p)$ and we test $H_0 : p = 0.5$ against $H_1 : p > 0.5$ at target significance level $\alpha = 0.05$ without randomization, the cumulative tail probabilities under $p = 0.5$ are $\mathbb{P}(X \ge 9) = 11/1024 \approx 0.01074$ and $\mathbb{P}(X \ge 8) = 56/1024 \approx 0.05469$. To respect the level constraint $\beta(0.5) \le 0.05$, a non-randomized rule must choose $\mathcal{C} = \{9, 10\}$, yielding a test of level $0.05$ whose actual size is $0.01074$. Conceding the gap between $0.01074$ and $0.05$ sacrifices substantial statistical power under $H_1$.
• Composite Null Hypotheses with Monotone Power: When testing $H_0 : \theta \le \theta_0$ versus $H_1 : \theta > \theta_0$, if the power function $\beta(\theta)$ is monotonically non-decreasing in $\theta$, then for every interior null point $\theta < \theta_0$ we have $\beta(\theta) \le \beta(\theta_0)$. Consequently, the supremum over $\Theta_0 = (-\infty, \theta_0]$ is attained precisely on the topological boundary $\partial \Theta_0 = \{\theta_0\}$:

This boundary reduction allows the investigator to calibrate the critical value exclusively at the boundary distribution $P_{\theta_0}$.

Consistency and Asymptotic Power Sequences

Let $\{\phi_n\}_{n=1}^\infty$ be a sequence of level-$\alpha$ tests based on samples of increasing size $n$. The test sequence is called consistent against the alternative $\Theta_1$ if its power converges to unity at every fixed alternative parameter point as the sample size grows without bound:

Consistency guarantees that for any fixed departure from $H_0$, however small, a sufficiently large sample will detect the departure with probability approaching certainty.

One of the deepest structural results in mathematical statistics is the exact mathematical equivalence—or duality—between the theory of hypothesis testing and the theory of confidence set estimation. Every family of hypothesis tests generates a confidence region, and every confidence region defines a family of hypothesis tests, with optimality properties translating directly across the two domains.

Theorem 1.1: The Test-Confidence Set Inversion Duality

Let $(\mathcal{X}, \mathcal{A}, \{P_\theta : \theta \in \Theta\})$ be a statistical model.

1. From Tests to Confidence Sets: Suppose that for every possible parameter point $\theta_0 \in \Theta$, we have a non-randomized level-$\alpha$ test for the point null hypothesis $H_0(\theta_0) : \theta = \theta_0$ with acceptance region $A(\theta_0) \subseteq \mathcal{X}$ and critical region $\mathcal{C}(\theta_0) = \mathcal{X} \setminus A(\theta_0)$, satisfying:

For each observed sample vector $\mathbf{x} \in \mathcal{X}$, define the subset $S(\mathbf{x}) \subseteq \Theta$ as the collection of all parameter values $\theta_0$ that are not rejected when $\mathbf{X} = \mathbf{x}$ is observed:

Then the random set $S(\mathbf{X})$ is a valid $(1 - \alpha)$ confidence set for $\theta$, satisfying:

2. From Confidence Sets to Tests: Conversely, suppose $S(\mathbf{X}) \subseteq \Theta$ is a $(1 - \alpha)$ confidence set for $\theta$ satisfying $\mathbb{P}_\theta(\theta \in S(\mathbf{X})) \ge 1 - \alpha$ for all $\theta \in \Theta$. For each $\theta_0 \in \Theta$, define the acceptance region:

Then the decision rule that rejects $H_0 : \theta = \theta_0$ if and only if $\mathbf{X} \notin A(\theta_0)$ (equivalently, if and only if $\theta_0 \notin S(\mathbf{X})$) is a valid level-$\alpha$ test of $H_0 : \theta = \theta_0$.

Formal Set-Theoretic Proof:

The proof rests on the fundamental equivalence of set membership across the Cartesian product space $\mathcal{X} \times \Theta$. Consider the subset $M \subseteq \mathcal{X} \times \Theta$ defined by:

For a fixed parameter $\theta \in \Theta$, the horizontal slice of $M$ is the acceptance region $A(\theta) \subseteq \mathcal{X}$. For a fixed sample realization $\mathbf{x} \in \mathcal{X}$, the vertical slice of $M$ is the confidence set $S(\mathbf{x}) \subseteq \Theta$. Therefore, for every pair $(\mathbf{x}, \theta) \in \mathcal{X} \times \Theta$, the logical equivalence holds identically:

Replacing the deterministic point $\mathbf{x}$ with the random sample vector $\mathbf{X} \sim P_\theta$, the two events $\{\theta \in S(\mathbf{X})\}$ and $\{\mathbf{X} \in A(\theta)\}$ are identical subsets of the underlying probability space! Consequently, for every $\theta \in \Theta$:

Since each test $H_0(\theta) : \vartheta = \theta$ has level $\alpha$, its Type I error satisfies $\beta_\theta(\theta) \le \alpha$, which immediately implies $\mathbb{P}_\theta(\theta \in S(\mathbf{X})) \ge 1 - \alpha$. If every test has exact size $\alpha$ so that $\mathbb{P}_\theta(\mathbf{X} \in A(\theta)) = 1 - \alpha$, then $S(\mathbf{X})$ has exact coverage probability $1 - \alpha$ everywhere in $\Theta$.

Uniformly Most Accurate (UMA) Confidence Bounds

The duality theorem extends beyond coverage probability to optimality criteria. How do we measure the accuracy of a confidence set $S(\mathbf{X})$? If $\theta$ is the true parameter and $\theta' \neq \theta$ is any false parameter value, an erroneous inclusion occurs whenever the confidence set $S(\mathbf{X})$ covers the false value $\theta'$. Using the fundamental equivalence $\theta' \in S(\mathbf{X}) \iff \mathbf{X} \in A(\theta')$, the probability of covering a false parameter value $\theta'$ when $\theta$ is true is:

Notice what this equation reveals: minimizing the probability that the confidence set $S(\mathbf{X})$ covers a false value $\theta'$ under the true parameter $\theta$ is mathematically identical to maximizing the power $\beta_{\theta'}(\theta)$ of the size-$\alpha$ test for $H_0 : \vartheta = \theta'$ against the alternative $H_1 : \vartheta = \theta$!

A $(1 - \alpha)$ confidence set $S^*(\mathbf{X})$ is defined to be Uniformly Most Accurate (UMA) if for every true parameter $\theta$ and false parameter $\theta'$, $\mathbb{P}_\theta(\theta' \in S^*(\mathbf{X}))$ is minimized among all $(1 - \alpha)$ confidence sets. By Theorem 1.1:

• Inverting a family of Uniformly Most Powerful (UMP) one-sided tests for $H_0 : \theta = \theta_0$ against $H_1 : \theta > \theta_0$ yields a Uniformly Most Accurate (UMA) lower confidence bound $[\underline{\theta}(\mathbf{X}), \infty)$ that minimizes $\mathbb{P}_\theta(\underline{\theta}(\mathbf{X}) \le \theta')$ for all $\theta' < \theta$.
• Inverting a family of Uniformly Most Powerful Unbiased (UMPU) two-sided tests yields a Uniformly Most Accurate Unbiased (UMAU) confidence interval $[\underline{\theta}(\mathbf{X}), \overline{\theta}(\mathbf{X})]$.

While a fixed-level Neyman-Pearson test reports only a binary decision at a pre-selected significance threshold $\alpha$, scientific reporting requires quantifying the exact strength of the empirical evidence against $H_0$. This role is fulfilled by the p-value (or observed significance level).

Formal Measure-Theoretic Definition of a Valid P-Value

Suppose we have a nested family of critical regions $\{\mathcal{C}_\alpha : \alpha \in (0, 1)\}$ for testing $H_0 : \theta \in \Theta_0$, where each $\mathcal{C}_\alpha$ is a valid level-$\alpha$ rejection region ($\sup_{\theta \in \Theta_0} \mathbb{P}_\theta(\mathbf{X} \in \mathcal{C}_\alpha) \le \alpha$) and the regions are monotonically nested so that $\mathcal{C}_\alpha \subseteq \mathcal{C}_{\alpha'}$ whenever $\alpha < \alpha'$. The p-value $p(\mathbf{X})$ is defined as the infimum of all significance levels $\alpha$ at which the observed data $\mathbf{X}$ would lead to rejection of $H_0$:

More generally, a measurable statistic $P(\mathbf{X}) : \mathcal{X} \to [0, 1]$ is defined to be a valid p-value for the null hypothesis $H_0 : \theta \in \Theta_0$ if for every $\theta \in \Theta_0$ and every threshold $u \in [0, 1]$:

If equality holds for some $\theta_0 \in \Theta_0$ across all $u \in [0, 1]$, then $P(\mathbf{X}) \sim \text{Uniform}(0, 1)$ under $P_{\theta_0}$, and the p-value is said to be exact. If $\mathbb{P}_\theta(P(\mathbf{X}) \le u) \le u$ with strict inequality for some $u$, the p-value is conservative (stochastically larger than a standard uniform random variable under $H_0$).

Theorem 1.2: Uniform Null Distribution of Continuous P-Values

Let $T(\mathbf{X})$ be a real-valued test statistic such that large values of $T(\mathbf{X})$ provide evidence against a simple null hypothesis $H_0 : \theta = \theta_0$ in favor of $H_1 : \theta \in \Theta_1$. Suppose the cumulative distribution function of $T(\mathbf{X})$ under $H_0$, denoted $F_0(t) = \mathbb{P}_{\theta_0}(T(\mathbf{X}) \le t)$, is continuous on $\mathbb{R}$, and define the upper-tail survival function $S_0(t) = 1 - F_0(t) = \mathbb{P}_{\theta_0}(T(\mathbf{X}) \ge t)$. Then the p-value statistic defined by:

where $\mathbf{X}^* \sim P_{\theta_0}$ is an independent copy of the sample under $H_0$, follows an exact continuous uniform distribution on $[0, 1]$ when $H_0$ is true:

Proof via the Probability Integral Transform:

Let $F_0^{-1}(q) = \inf\{t \in \mathbb{R} : F_0(t) \ge q\}$ denote the generalized quantile function of $F_0$. Because $F_0$ is continuous, $F_0(F_0^{-1}(q)) = q$ for all $q \in (0, 1)$, and for any $t \in \mathbb{R}$ and $q \in (0, 1)$, the inequality $F_0(t) \ge q$ holds if and only if $t \ge F_0^{-1}(q)$. For any $u \in (0, 1)$, evaluate the cumulative distribution function of $P(\mathbf{X}) = 1 - F_0(T(\mathbf{X}))$ under $H_0$:

Using the quantile equivalence and the continuity of $T(\mathbf{X})$ under $H_0$:

Thus $\mathbb{P}_{\theta_0}(P(\mathbf{X}) \le u) = u$ for all $u \in [0, 1]$, proving that $P(\mathbf{X}) \sim \text{Uniform}(0, 1)$ under $H_0$.

Distribution of P-Values Under the Alternative Hypothesis and Discrete Randomization

What is the probability distribution of the p-value $P(\mathbf{X}) = 1 - F_0(T(\mathbf{X}))$ when the alternative hypothesis $H_1 : \theta = \theta_1$ is true? Let $F_1(t) = \mathbb{P}_{\theta_1}(T(\mathbf{X}) \le t)$ denote the cumulative distribution function of $T(\mathbf{X})$ under the alternative $\theta_1$. Then for any $u \in (0, 1)$, the cumulative distribution function of $P(\mathbf{X})$ under $H_1$ is:

Notice that $F_0^{-1}(1 - u)$ is the exact critical value $c_u$ of a size-$u$ upper-tail test, so $G_1(u) = \beta_{\theta_1}(u)$ is precisely the power of the size-$u$ test against $\theta_1$! Whenever the test is unbiased ($\beta_{\theta_1}(u) > u$), we have $G_1(u) > u$ for all $u \in (0, 1)$, meaning that $P(\mathbf{X})$ under $H_1$ is stochastically smaller than $\text{Uniform}(0, 1)$, concentrating probability mass heavily near zero.

When $T(\mathbf{X})$ is a discrete random variable taking values $\{t_1 < t_2 < \dots\}$, the standard tail probability $p_{\text{disc}}(t_k) = \mathbb{P}_{\theta_0}(T \ge t_k)$ only assumes discrete jump values $u_k = \mathbb{P}_{\theta_0}(T \ge t_k)$ and is strictly conservative between jumps. To obtain an exact $\text{Uniform}(0, 1)$ p-value in discrete models, one defines the randomized p-value:

And in applied reporting where auxiliary randomization is undesirable, Lancaster's mid-p-value replaces the random $U$ with its expectation $\mathbb{E}[U] = 1/2$, yielding $p_{\text{mid}}(t) = \mathbb{P}_{\theta_0}(T > t) + \frac{1}{2} \mathbb{P}_{\theta_0}(T = t)$.

In contemporary scientific domains such as genomics, neuroimaging, high-energy astrophysics, and quantitative finance, investigators routinely test $m$ statistical hypotheses $H_{0,1}, H_{0,2}, \dots, H_{0,m}$ simultaneously from the same dataset (where $m$ may range from tens to millions). If each individual hypothesis $H_{0,i}$ is tested separately at significance level $\alpha = 0.05$, the probability of committing at least one Type I error across the collection of $m$ tests escalates rapidly. For $m$ mutually independent true null hypotheses, the probability of making at least one false rejection is: $$\mathbb{P}(\text{At least one false rejection}) = 1 - (1 - \alpha)^m$$ Even for a modest family of $m = 20$ independent true null hypotheses at $\alpha = 0.05$, this probability reaches $1 - (0.95)^{20} \approx 0.6415$ ($64.15\%$), and the expected number of false discoveries is $m\alpha = 1$. Controlling false positives across simultaneous tests requires rigorous multiple testing error metrics. ### Classification of Multiple Testing Outcomes Suppose $m$ hypotheses are tested simultaneously, of which $m_0$ are truly null (the set $\mathcal{M}_0 \subseteq \{1, \dots, m\}$ with $|\mathcal{M}_0| = m_0$ unknown) and $m_1 = m - m_0$ are truly alternative. Let $R$ denote the total observable number of rejected hypotheses, and let $V$ denote the unobservable number of false rejections (Type I errors): ### Family-Wise Error Rate (FWER): Bonferroni and Holm Step-Down Procedures The **Family-Wise Error Rate (FWER)** is defined as the probability of committing one or more false rejections across the entire family of $m$ tests: $$\text{FWER} = \mathbb{P}(V \ge 1)$$ A multiple testing procedure controls FWER in the **weak sense** if $\mathbb{P}(V \ge 1) \le \alpha$ only under the complete null configuration $m_0 = m$, and controls FWER in the **strong sense** if $\mathbb{P}(V \ge 1) \le \alpha$ for *every* possible subset of true nulls $\mathcal{M}_0 \subseteq \{1, \dots, m\}$. 1. **The Bonferroni Procedure**: Reject $H_{0,i}$ if its p-value satisfies $P_i \le \alpha / m$. To prove strong FWER control under **arbitrary dependence** among the p-values, apply Boole's union bound over the set of true null hypotheses $\mathcal{M}_0$: $$\text{FWER} = \mathbb{P}\left(\bigcup_{i \in \mathcal{M}_0} \left\{P_i \le \frac{\alpha}{m}\right\}\right) \le \sum_{i \in \mathcal{M}_0} \mathbb{P}\left(P_i \le \frac{\alpha}{m}\right) \le \sum_{i \in \mathcal{M}_0} \frac{\alpha}{m} = \frac{m_0}{m} \alpha \le \alpha$$ 2. **Holm's Step-Down Procedure (1979)**: Because Bonferroni divides by $m$ even when many hypotheses are clearly false ($m_0 < m$), Sture Holm developed a uniformly more powerful step-down method that still guarantees strong FWER control under arbitrary dependence. Order the observed p-values from smallest to largest: $P_{(1)} \le P_{(2)} \le \dots \le P_{(m)}$, with associated hypotheses $H_{0,(1)}, \dots, H_{0,(m)}$. For each step $k = 1, 2, \dots, m$, compare $P_{(k)}$ against the threshold $\frac{\alpha}{m - k + 1}$. Let $k^*$ be the smallest index such that $P_{(k^*)} > \frac{\alpha}{m - k^* + 1}$. Reject all hypotheses $H_{0,(1)}, \dots, H_{0,(k^*-1)}$ and fail to reject $H_{0,(k^*)}, \dots, H_{0,(m)}$. - *Proof of Strong FWER Control*: Suppose $V \ge 1$, and let $j$ be the smallest ordered index among all rejected hypotheses that actually belongs to the true null set $\mathcal{M}_0$. Since at most $m - m_0$ false nulls could have preceded $j$ in the ordering, the rank $j$ must satisfy $j \le m - m_0 + 1$, which implies $m - j + 1 \ge m_0$. Because $H_{0,(j)}$ was rejected, we must have $P_{(j)} \le \frac{\alpha}{m - j + 1} \le \frac{\alpha}{m_0}$. Thus the event $\{V \ge 1\}$ implies that at least one true null hypothesis $i \in \mathcal{M}_0$ has $P_i \le \alpha / m_0$. By Boole's inequality, $\mathbb{P}(V \ge 1) \le \sum_{i \in \mathcal{M}_0} \mathbb{P}(P_i \le \alpha / m_0) \le m_0 (\alpha / m_0) = \alpha$. ### False Discovery Rate (FDR) and the Benjamini-Hochberg Step-Up Procedure When $m$ is in the thousands, guarding against even a single false rejection via FWER destroys statistical power. Yoav Benjamini and Yosef Hochberg (1995) revolutionized large-scale inference by introducing the **False Discovery Rate (FDR)**, defined as the expected proportion of false discoveries among all rejected hypotheses (defining $V / R = 0$ when $R = 0$): $$\text{FDR} = \mathbb{E}\left[\frac{V}{\max(R, 1)}\right] = \mathbb{E}\left[\frac{V}{R} \,\mathbf{1}_{\{R > 0\}}\right]$$ In the **Benjamini-Hochberg (BH) Step-Up Procedure** at target FDR level $q \in (0, 1)$, we order the p-values $P_{(1)} \le P_{(2)} \le \dots \le P_{(m)}$ and find the **largest** index $\hat{k}$ satisfying: $$\hat{k} = \max\left\{k \in \{1, \dots, m\} : P_{(k)} \le \frac{k}{m} q\right\}$$ If such a $\hat{k}$ exists, we reject all hypotheses $H_{0,(1)}, H_{0,(2)}, \dots, H_{0,(\hat{k})}$; otherwise we reject none. Benjamini and Hochberg proved that when the test statistics corresponding to the true null hypotheses $\mathcal{M}_0$ are mutually independent (or satisfy positive regression dependence on a subset, PRDS), the BH procedure satisfies the exact mathematical identity (for continuous uniform null p-values): $$\text{FDR} = \frac{m_0}{m} q \le q$$ Because the BH threshold $\frac{k}{m} q$ scales linearly with the rank $k$ rather than remaining fixed at $\alpha / m$, it achieves dramatically higher statistical power in high-dimensional screening studies while rigorously bounding the expected fraction of false leads.