Sequential Hypothesis Testing: Wald's SPRT, Stopping Boundaries, OC & ASN Functions

To formalize sequential hypothesis testing within measure-theoretic probability, let $X_1, X_2, \dots$ be an infinite sequence of independent and identically distributed (i.i.d.) random variables defined on a probability space $(\Omega, \mathcal{F}, \mathbb{P}_\theta)$, where each $X_i$ possesses probability density function (or probability mass function in the discrete case) $f(x; \theta)$ with respect to a $\sigma$-finite dominating measure $\mu$. Let $\{\mathcal{F}_m\}_{m \ge 1}$ denote the natural filtration generated by the observation sequence, where $\mathcal{F}_m = \sigma(X_1, X_2, \dots, X_m)$ represents the cumulative information available up to and including stage $m$, with $\mathcal{F}_0 = \{\emptyset, \Omega\}$ representing the trivial initial $\sigma$-algebra prior to sampling. A sequential test is formally defined as an ordered pair $(N, \delta_N)$ consisting of two distinct mathematical components: 1. **A Stopping Rule (Stopping Time) $N$**: A random variable $N : \Omega \to \{0, 1, 2, \dots\} \cup \{\infty\}$ is a valid stopping time with respect to the filtration $\{\mathcal{F}_m\}$ if, for every non-negative integer $m \ge 0$, the event $\{N = m\}$ belongs to $\mathcal{F}_m$. Equivalently, the indicator variable $\mathbb{I}(N \le m)$ is a measurable function of $(X_1, \dots, X_m)$ alone. This measurability condition guarantees that the decision to terminate sampling at step $m$ depends strictly on the historical data observed up to step $m$ and involves no clairvoyant foresight of future unrealized observations $X_{m+1}, X_{m+2}, \dots$. 2. **A Terminal Decision Rule $\delta_N$**: A sequence of measurable decision functions $\{\delta_m(X_1, \dots, X_m)\}_{m \ge 1}$, where each $\delta_m : \mathbb{R}^m \to \{0, 1\}$ maps the observed $m$-dimensional sample path into a binary decision upon termination ($0$ denoting acceptance of $H_0$, and $1$ denoting rejection of $H_0$ in favor of $H_1$). Consider testing a simple null hypothesis $H_0 : \theta = \theta_0$ against a simple alternative hypothesis $H_1 : \theta = \theta_1$, where $\theta_0 \ne \theta_1$. Denote the null density by $f_0(x) = f(x; \theta_0)$ and the alternative density by $f_1(x) = f(x; \theta_1)$. At any fixed stage $m$, the Neyman-Pearson Fundamental Lemma dictates that the most powerful test is governed by the cumulative likelihood ratio: $$\Lambda_m(X_1, \dots, X_m) = \frac{L_m(\theta_1)}{L_m(\theta_0)} = \prod_{i=1}^m \frac{f_1(X_i)}{f_0(X_i)}$$ By taking the natural logarithm of $\Lambda_m$, the multiplicative likelihood ratio transforms into an additive random walk. Specifically, if we define the single-observation log-likelihood ratio increment (or weight of evidence) as: $$Z_i = \ln \left[ \frac{f_1(X_i)}{f_0(X_i)} \right], \quad i = 1, 2, \dots$$ then the cumulative log-likelihood ratio at stage $m$ is simply the partial sum of i.i.d. random variables: $$S_m = \ln \Lambda_m = \sum_{i=1}^m Z_i = S_{m-1} + Z_i, \quad \text{with } S_0 = 0$$ Abraham Wald's **Sequential Probability Ratio Test (SPRT)** with stopping boundaries $(B, A)$, where $0 < B < 1 < A < \infty$, operates as follows. At each sampling stage $m = 1, 2, \dots$, we compute $\Lambda_m$ (or equivalently $S_m = \ln \Lambda_m$ against the logarithmic thresholds $b = \ln B < 0 < a = \ln A$) and execute one of three rules: - **Accept $H_0$**: If $\Lambda_m \le B$ (equivalently, $S_m \le b = \ln B$), terminate the sampling process immediately at step $N = m$ and decide in favor of $H_0$ ($\delta_N = 0$). - **Reject $H_0$ (Accept $H_1$)**: If $\Lambda_m \ge A$ (equivalently, $S_m \ge a = \ln A$), terminate the sampling process immediately at step $N = m$ and decide in favor of $H_1$ ($\delta_N = 1$). - **Continue Sampling**: If $B < \Lambda_m < A$ (equivalently, $b < S_m < a$), neither hypothesis has accumulated sufficient evidentiary weight; draw observation $X_{m+1}$ and advance to stage $m+1$. Mathematically, the stopping time $N$ of Wald's SPRT is the first exit time of the discrete-time random walk $\{S_m\}_{m \ge 1}$ from the open continuation interval $(b, a)$: $$N = \inf \left\{ m \ge 1 : \Lambda_m \notin (B, A) \right\} = \inf \left\{ m \ge 1 : S_m \le b \text{ or } S_m \ge a \right\}$$ This formulation reveals a deep structural bridge between statistical inference and stochastic processes: executing an SPRT is isomorphic to solving the classic **Gambler's Ruin** or two-barrier first-passage problem for a one-dimensional random walk with drift $\mu_Z(\theta) = \mathbb{E}_\theta[Z_i]$. Under the null hypothesis $H_0$, by Jensen's inequality applied to the strictly concave logarithm function (assuming $\mathbb{P}_{\theta_0}(f_1(X) \ne f_0(X)) > 0$), the expected increment is strictly negative: $$\mathbb{E}_{\theta_0}[Z_i] = \mathbb{E}_{\theta_0}\left[\ln \frac{f_1(X_i)}{f_0(X_i)}\right] < \ln \mathbb{E}_{\theta_0}\left[\frac{f_1(X_i)}{f_0(X_i)}\right] = \ln \int f_1(x) \, d\mu(x) = \ln(1) = 0$$ Notice that $\mathbb{E}_{\theta_0}[Z_i] = -D_{\text{KL}}(f_0 \parallel f_1) < 0$, where $D_{\text{KL}}(f_0 \parallel f_1)$ is the Kullback-Leibler divergence of $f_1$ from $f_0$. Conversely, under $H_1$, $\mathbb{E}_{\theta_1}[Z_i] = +D_{\text{KL}}(f_1 \parallel f_0) > 0$. Thus, when $H_0$ is true, the random walk $S_m$ exhibits negative drift downward toward the lower absorption barrier $b = \ln B$; when $H_1$ is true, $S_m$ exhibits positive drift upward toward the upper absorption barrier $a = \ln A$.

In practical statistical design, an investigator specifies the desired Type I error probability $\alpha = \mathbb{P}_{\theta_0}(\text{Reject } H_0)$ and Type II error probability $\beta = \mathbb{P}_{\theta_1}(\text{Accept } H_0)$ in advance, and seeks the exact numerical values of the stopping thresholds $A$ and $B$ that achieve $(\alpha, \beta)$. Unlike fixed-sample tests where exact quantile inversion of a single distribution suffices, exact determination of $A$ and $B$ in a discrete-time random walk requires tracking the distribution of the excess over the boundary at termination. Nevertheless, Abraham Wald discovered a pair of universal, distribution-free inequalities linking $(A, B)$ and $(\alpha, \beta)$ that hold across all probability distributions without any parametric assumptions.

Let $\mathcal{R}_m \subset \mathbb{R}^m$ denote the subset of sample paths $(x_1, \dots, x_m)$ for which the SPRT terminates at stage $N = m$ with rejection of $H_0$ ($\Lambda_m \ge A$), and let $\mathcal{A}_m \subset \mathbb{R}^m$ denote the subset of sample paths for which the SPRT terminates at stage $N = m$ with acceptance of $H_0$ ($\Lambda_m \le B$). Because the stopping events at distinct stages $m = 1, 2, \dots$ are mutually exclusive (a path can stop for the first time at most once), the overall probability of rejecting $H_0$ when $\theta = \theta_1$ is the countable sum of disjoint stage-wise rejection probabilities. Assuming the test terminates with probability one ($\mathbb{P}_\theta(N < \infty) = 1$, which we prove rigorously in Section 5.3), the probability of rejecting $H_0$ under $H_1$ is $1 - \beta$, while the probability of rejecting $H_0$ under $H_0$ is $\alpha$.

Theorem 5.1 (Wald's Fundamental Stopping Boundary Inequalities)

For any Sequential Probability Ratio Test with stopping boundaries $(B, A)$ that terminates with probability one under both $H_0$ and $H_1$, the true Type I error probability $\alpha$ and Type II error probability $\beta$ satisfy the exact inequalities:

Equivalently, solving for the error rates in terms of the boundaries yields:

Furthermore, the sum of the realized error probabilities if one sets the boundaries to the nominal Wald values $A_W = (1 - \beta_0)/\alpha_0$ and $B_W = \beta_0/(1 - \alpha_0)$ satisfies $\alpha + \beta \le \alpha_0 + \beta_0$.

Proof:

Consider the event of rejecting $H_0$, which is the disjoint union $\bigcup_{m=1}^\infty \mathcal{R}_m$. On every sample path $(x_1, \dots, x_m) \in \mathcal{R}_m$, the stopping rule requires that the likelihood ratio at termination satisfies $\Lambda_m(x_1, \dots, x_m) \ge A$, which is equivalent to the pointwise density inequality:

Integrating (or summing) both sides of this inequality over the rejection region $\mathcal{R}_m$ with respect to the product measure $d\mu^m$ and summing over all stages $m = 1, 2, \dots, \infty$, we obtain:

Dividing both sides of $1 - \beta \ge A \alpha$ by $\alpha > 0$ immediately yields the upper bound $A \le \frac{1 - \beta}{\alpha}$.

By completely symmetric reasoning, consider the event of accepting $H_0$, which is the disjoint union $\bigcup_{m=1}^\infty \mathcal{A}_m$. On every sample path $(x_1, \dots, x_m) \in \mathcal{A}_m$, termination at the lower boundary requires $\Lambda_m(x_1, \dots, x_m) \le B$, or equivalently:

Integrating over $\mathcal{A}_m$ and summing from $m = 1$ to $\infty$ yields:

Dividing by $1 - \alpha > 0$ establishes $B \ge \frac{\beta}{1 - \alpha}$.

Now suppose an investigator specifies target nominal error rates $(\alpha_0, \beta_0)$ and sets the boundaries using Wald's Boundary Approximations:

Let $\alpha_{\text{true}}$ and $\beta_{\text{true}}$ denote the actual Type I and Type II error probabilities induced by these boundaries $(B_W, A_W)$. Applying the two inequalities proven above to $(\alpha_{\text{true}}, \beta_{\text{true}})$ gives:

Adding these two cross-multiplied inequalities together yields:

Canceling the identical cross-product terms $-\alpha_{\text{true}}\beta_0 - \beta_{\text{true}}\alpha_0$ from both sides produces the celebrated conservation bound:

This proves that at least one of the true error probabilities $\alpha_{\text{true}}$ or $\beta_{\text{true}}$ must be less than or equal to its nominal target, and neither error rate can exceed its target by more than a negligible fraction $\alpha_0 \beta_0$: specifically, $\alpha_{\text{true}} \le \frac{\alpha_0}{1 - \beta_0} \approx \alpha_0(1 + \beta_0)$ and $\beta_{\text{true}} \le \frac{\beta_0}{1 - \alpha_0} \approx \beta_0(1 + \alpha_0)$. For instance, if $\alpha_0 = 0.05$ and $\beta_0 = 0.05$, then $\alpha_{\text{true}} \le 0.05/0.95 = 0.0526$.

Why are Wald's inequalities strict inequalities in discrete time rather than exact equalities? Because the random walk $S_m$ evolves in discrete steps $Z_m$, termination at stage $N$ almost always involves overshooting the boundary (the excess over the boundary): at the moment of rejection, $S_N$ does not hit $a = \ln A$ exactly on the nose, but rather jumps from $S_{N-1} < a$ to $S_N = a + R_+ > a$, where $R_+ = S_N - a > 0$ is the overshoot. In the continuous-time limit where individual increments $Z_i$ have infinitesimal variance (as when $\theta_1$ is close to $\theta_0$, leading to Brownian motion sample paths with continuous trajectories), the overshoot $R_+$ vanishes identically, and Wald's approximations $A = (1-\beta)/\alpha$ and $B = \beta/(1-\alpha)$ become exact equalities. Due to this slight overshoot in discrete time, using $(B_W, A_W)$ actually produces a slightly conservative test whose true error probabilities $\alpha_{\text{true}}$ and $\beta_{\text{true}}$ are typically smaller than $\alpha_0$ and $\beta_0$, at the slight expense of increasing the expected sample size by one or two observations.

A foundational prerequisite for any sequential procedure is that it must eventually terminate: if there were a positive probability $\mathbb{P}_\theta(N = \infty) > 0$ that the random walk $S_m$ wandered forever inside the open strip $(b, a)$ without ever crossing either threshold, the experiment could continue indefinitely. Charles Stein (1946) proved a remarkably strong theorem showing not only that $\mathbb{P}_\theta(N < \infty) = 1$ for every distribution where $Z_i$ is not identically zero, but also that the stopping time $N$ possesses an exponentially decaying tail, guaranteeing that all moments $\mathbb{E}_\theta[N^k]$ ($k = 1, 2, \dots$) and the moment generating function of $N$ around zero are strictly finite.

Theorem 5.2 (Stein's Lemma on Exponential Tail Decay and Almost-Sure Termination of the SPRT)

Let $Z_1, Z_2, \dots$ be i.i.d. random variables under probability measure $\mathbb{P}_\theta$ such that $\mathbb{P}_\theta(Z_1 = 0) < 1$. Let $N = \inf\{m \ge 1 : S_m \notin (b, a)\}$ be the stopping time of an SPRT with finite logarithmic boundaries $-\infty < b < 0 < a < \infty$. Then:

1. There exist constants $C > 0$ and $\rho \in (0, 1)$ such that for all integers $n \ge 1$:

1. Consequently, the SPRT terminates in a finite number of steps with probability one:

1. All positive integer moments of the stopping time are finite: $\mathbb{E}_\theta[N^k] < \infty$ for all $k \ge 1$, and there exists $t_0 > 0$ such that the moment generating function $\mathbb{E}_\theta[e^{tN}] < \infty$ for all $t < t_0$.

Proof:

Let $L = a - b > 0$ denote the total width of the open continuation interval $(b, a)$. Because $\mathbb{P}_\theta(Z_1 = 0) < 1$, the distribution of $Z_1$ is non-degenerate at zero; hence either $\mathbb{P}_\theta(Z_1 > 0) > 0$ or $\mathbb{P}_\theta(Z_1 < 0) > 0$ (or both). Without loss of generality, suppose $\mathbb{P}_\theta(Z_1 > 0) > 0$. Then there exists a small positive threshold $\delta > 0$ and a strictly positive probability $\epsilon > 0$ such that:

Choose a fixed positive integer $r$ sufficiently large so that $r \delta > L = a - b$; specifically, let $r = \lfloor (a - b)/\delta \rfloor + 1$. Consider a block of $r$ consecutive increments $Z_{j+1}, Z_{j+2}, \dots, Z_{j+r}$ starting after any arbitrary stage $j \ge 0$. If each of these $r$ independent increments satisfies $Z_{j+i} \ge \delta$ for $i = 1, \dots, r$, then their block sum satisfies:

Because $Z_{j+1}, \dots, Z_{j+r}$ are independent and identically distributed, the probability of this block event is strictly positive:

Now observe that if the random walk is inside the continuation strip $(b, a)$ at stage $j$ (so $b < S_j < a$), and the subsequent $r$-step block sum exceeds $a - b$, then at stage $j + r$ the cumulative sum must satisfy:

Therefore, it is mathematically impossible for both $S_j \in (b, a)$ and $S_{j+r} \in (b, a)$ to occur simultaneously when $\sum_{i=1}^r Z_{j+i} > a - b$! Contrapositively, for the process to remain inside the continuation strip throughout the entire block from $j+1$ to $j+r$, the block sum cannot exceed $a - b$:

Now consider any arbitrary sample size $n \ge 1$, and divide the first $n$ observations into $k = \lfloor n / r \rfloor$ disjoint, independent blocks of length $r$ each. For the stopping time $N$ to exceed $n$, the random walk $S_m$ must remain strictly inside $(b, a)$ at the end of every single block $m = r, 2r, \dots, kr$. By the independence of disjoint blocks of i.i.d. increments:

Since $k = \lfloor n / r \rfloor > \frac{n}{r} - 1$, we can write $(1 - \epsilon^r)^k \le (1 - \epsilon^r)^{-1} \left[ (1 - \epsilon^r)^{1/r} \right]^n$. Defining the constants $C = (1 - \epsilon^r)^{-1} > 0$ and $\rho = (1 - \epsilon^r)^{1/r} \in (0, 1)$, we obtain the geometric tail bound:

Taking the limit as $n \to \infty$, since $0 < \rho < 1$:

Finally, for any $t > 0$ such that $e^t \rho < 1$ (i.e., $t < t_0 = -\ln \rho > 0$), the moment generating function of $N$ is bounded by a convergent geometric series:

Since the moment generating function $\mathbb{E}_\theta[e^{tN}]$ is finite in a neighborhood of $t = 0$, all moments $\mathbb{E}_\theta[N^k]$ are finite. This completes the proof of Stein's Lemma.

To evaluate the performance of a sequential test across the entire parameter space $\Theta$—not merely at the two design points $\theta_0$ and $\theta_1$—we require the Operating Characteristic (OC) function, denoted $L(\theta)$, which gives the probability of accepting the null hypothesis $H_0$ as a function of the true parameter $\theta$:

Note that the power function of the sequential test is simply the complement of the OC function: $\text{Power}(\theta) = 1 - L(\theta) = \mathbb{P}_\theta(S_N \ge a)$. To compute $L(\theta)$ analytically, Abraham Wald proved one of the most powerful martingale identities in stochastic processes: Wald's Fundamental Identity in sequential analysis.

Let $M_\theta(t) = \mathbb{E}_\theta[e^{t Z_1}]$ denote the moment generating function (MGF) of a single log-likelihood ratio increment $Z_1 = \ln[f_1(X_1)/f_0(X_1)]$ when the true parameter is $\theta$. Define the discrete-time stochastic process $\{W_m(t)\}_{m \ge 0}$ by:

Observe that $\{W_m(t)\}_{m \ge 0}$ is a positive martingale with respect to the filtration $\{\mathcal{F}_m\}$, because for every $m \ge 0$:

By Doob's Optional Stopping Theorem ( combined with Stein's exponential tail bound on $N$ and the uniform boundedness of $S_{m \wedge N}$ inside $(b, a)$ prior to stopping), the expected value of the martingale at the stopping time $N$ equals its initial value $W_0(t) = 1$.

Theorem 5.3 (Wald's Fundamental Identity of Sequential Analysis)

Let $Z_1, Z_2, \dots$ be i.i.d. increments with moment generating function $M_\theta(t) = \mathbb{E}_\theta[e^{t Z_1}]$ finite in an open interval containing $t$, and let $N$ be the stopping time of an SPRT with finite boundaries $(b, a)$. Then for all $t$ in the domain of $M_\theta(t)$ such that $|M_\theta(t)| \ge 1$ (or within a neighborhood of the roots of $M_\theta(t) = 1$):

How does Wald's Fundamental Identity unlock the Operating Characteristic function $L(\theta)$? Suppose that for a given parameter value $\theta$ (with $\mathbb{E}_\theta[Z_1] \ne 0$), we can find a non-zero real root $t = h(\theta) \ne 0$ of the characteristic equation:

Because $M_\theta(t)$ is strictly convex ($M_\theta''(t) = \mathbb{E}_\theta[Z_1^2 e^{t Z_1}] > 0$), satisfies $M_\theta(0) = 1$, and has derivative $M_\theta'(0) = \mathbb{E}_\theta[Z_1] \ne 0$ at the origin, convexity guarantees that there exists at most one non-zero real root $h(\theta) \ne 0$ such that $M_\theta(h(\theta)) = 1$ (specifically, if $\mathbb{E}_\theta[Z_1] < 0$, then $h(\theta) > 0$; if $\mathbb{E}_\theta[Z_1] > 0$, then $h(\theta) < 0$). Notice that at the two design parameters $\theta_0$ and $\theta_1$, we can verify the root $h(\theta)$ by direct inspection:

• When $\theta = \theta_0$: $\mathbb{E}_{\theta_0}\left[\left(\frac{f_1(X)}{f_0(X)}\right)^1\right] = \int \frac{f_1(x)}{f_0(x)} f_0(x) \, d\mu(x) = \int f_1(x) \, d\mu(x) = 1$, so $h(\theta_0) = +1$.
• When $\theta = \theta_1$: $\mathbb{E}_{\theta_1}\left[\left(\frac{f_1(X)}{f_0(X)}\right)^{-1}\right] = \int \frac{f_0(x)}{f_1(x)} f_1(x) \, d\mu(x) = \int f_0(x) \, d\mu(x) = 1$, so $h(\theta_1) = -1$.

Now substitute $t = h(\theta) \ne 0$ into Wald's Fundamental Identity. Since $M_\theta(h(\theta)) = 1$, the random factor $[M_\theta(h(\theta))]^{-N} = 1^{-N} = 1$ drops out completely, leaving the remarkably clean identity:

Splitting this expectation across the two mutually exclusive terminal events—acceptance of $H_0$ ($S_N \le b$, which occurs with probability $L(\theta)$) and rejection of $H_0$ ($S_N \ge a$, which occurs with probability $1 - L(\theta)$)—and neglecting the boundary overshoot (so that $S_N \approx b = \ln B$ upon acceptance and $S_N \approx a = \ln A$ upon rejection), we obtain:

Solving this linear equation for $L(\theta)$ gives Wald's Operating Characteristic (OC) Formula:

What happens at the singular intermediate parameter value $\theta = \theta^*$ where the drift is zero ($\mathbb{E}_{\theta^*}[Z_1] = 0$, which corresponds to the confluent root $h(\theta^*) = 0$)? Applying L'Hôpital's rule to the OC formula as $h \to 0$:

This provides a complete parametric representation of the OC curve $(\theta(h), L(h))$ across the entire real line!

In sequential analysis, the sample size $N$ is a random variable whose expectation $\mathbb{E}_\theta[N]$ is called the **Average Sample Number (ASN) function**. Just as the OC curve characterizes the statistical accuracy of the SPRT across $\theta$, the ASN curve $\mathbb{E}_\theta[N]$ characterizes its sampling efficiency (expected cost or duration) as a function of the true parameter $\theta$. To derive a closed-form expression for $\mathbb{E}_\theta[N]$, we differentiate Wald's Fundamental Identity with respect to $t$ at $t = 0$ (or equivalently apply Wald's First Equation for stopped random walks). ### Theorem 5.4 (Wald's Equation and the ASN Function) Let $Z_1, Z_2, \dots$ be i.i.d. random variables with finite mean $\mathbb{E}_\theta[Z_1]$ and finite variance $\text{Var}_\theta(Z_1)$, and let $N$ be a stopping time with $\mathbb{E}_\theta[N] < \infty$. Then: 1. **Wald's First Equation (Non-zero Drift, $\mathbb{E}_\theta[Z_1] \ne 0$)**: $$\mathbb{E}_\theta[S_N] = \mathbb{E}_\theta[N] \cdot \mathbb{E}_\theta[Z_1] \quad \implies \quad \mathbb{E}_\theta[N] = \frac{\mathbb{E}_\theta[S_N]}{\mathbb{E}_\theta[Z_1]}$$ 2. **Wald's Second Equation (Zero Drift, $\mathbb{E}_{\theta^*}[Z_1] = 0$)**: $$\mathbb{E}_{\theta^*}\left[ S_N^2 \right] = \mathbb{E}_{\theta^*}[N] \cdot \mathbb{E}_{\theta^*}\left[ Z_1^2 \right] \quad \implies \quad \mathbb{E}_{\theta^*}[N] = \frac{\mathbb{E}_{\theta^*}[S_N^2]}{\mathbb{E}_{\theta^*}[Z_1^2]}$$ #### Proof via Differentiation of Wald's Fundamental Identity: Differentiating $\mathbb{E}_\theta\left[ e^{t S_N} (M_\theta(t))^{-N} \right] = 1$ with respect to $t$ under the expectation sign (justified by Stein's exponential tail bound) gives: $$\mathbb{E}_\theta \left[ S_N e^{t S_N} (M_\theta(t))^{-N} - N e^{t S_N} (M_\theta(t))^{-N-1} M_\theta'(t) \right] = 0$$ Evaluating at $t = 0$, where $M_\theta(0) = 1$ and $M_\theta'(0) = \mathbb{E}_\theta[Z_1]$, immediately yields $\mathbb{E}_\theta[S_N - N \mathbb{E}_\theta[Z_1]] = 0$, proving Wald's First Equation $\mathbb{E}_\theta[S_N] = \mathbb{E}_\theta[N]\mathbb{E}_\theta[Z_1]$. If $\mathbb{E}_{\theta^*}[Z_1] = 0$, differentiating a second time at $t = 0$ where $M_{\theta^*}'(0) = 0$ and $M_{\theta^*}''(0) = \mathbb{E}_{\theta^*}[Z_1^2]$ yields $\mathbb{E}_{\theta^*}[S_N^2 - N \mathbb{E}_{\theta^*}[Z_1^2]] = 0$, establishing Wald's Second Equation. Under Wald's small-overshoot approximation, the terminal log-likelihood ratio $S_N$ is approximated as a binary random variable taking value $b = \ln B$ with probability $L(\theta)$ and value $a = \ln A$ with probability $1 - L(\theta)$. Thus: $$\mathbb{E}_\theta[S_N] \approx L(\theta) \ln B + \left[ 1 - L(\theta) \right] \ln A = L(\theta) b + \left[ 1 - L(\theta) \right] a$$ Substituting this into Wald's First and Second Equations yields the universal **ASN Formula**: $$\mathbb{E}_\theta[N] \approx \begin{cases} \dfrac{L(\theta) \ln B + [1 - L(\theta)] \ln A}{\mathbb{E}_\theta[Z_1]} & \text{if } \mathbb{E}_\theta[Z_1] \ne 0 \\[12pt] \dfrac{-\ln A \ln B}{\mathbb{E}_{\theta^*}[Z_1^2]} = \dfrac{|b| a}{\text{Var}_{\theta^*}(Z_1)} & \text{if } \mathbb{E}_{\theta^*}[Z_1] = 0 \end{cases}$$ In particular, at the null hypothesis $\theta = \theta_0$ where $L(\theta_0) = 1 - \alpha$, and at the alternative hypothesis $\theta = \theta_1$ where $L(\theta_1) = \beta$: $$\mathbb{E}_{\theta_0}[N] \approx \frac{(1 - \alpha) \ln B + \alpha \ln A}{\mathbb{E}_{\theta_0}[Z_1]}, \qquad \mathbb{E}_{\theta_1}[N] \approx \frac{\beta \ln B + (1 - \beta) \ln A}{\mathbb{E}_{\theta_1}[Z_1]}$$ Notice the geometric shape of the ASN curve $\mathbb{E}_\theta[N]$ as a function of $\theta$: it is low at $\theta_0$ and $\theta_1$ (and approaches $1$ as $\theta \to \pm \infty$ where drift is huge), but achieves its **maximum peak** near the zero-drift indifference point $\theta = \theta^*$ where $\mathbb{E}_{\theta^*}[Z_1] = 0$. At $\theta^*$, the random walk has no drift and leaves $(b, a)$ purely by diffusive fluctuations, scaling proportionally to $|b|a / \sigma_Z^2$. Let us now catalog the exact log-likelihood increment $Z_i$, non-zero root condition $h(\theta)$, and sequential decision boundaries for the four canonical parametric families: 1. **Normal Mean with Known Variance**: $X_i \stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(\theta, \sigma^2)$ for testing $H_0 : \theta = \theta_0$ vs. $H_1 : \theta = \theta_1$ ($\theta_1 > \theta_0$). - Log-likelihood increment: $Z_i = \frac{\theta_1 - \theta_0}{\sigma^2} X_i - \frac{\theta_1^2 - \theta_0^2}{2\sigma^2} = \frac{\theta_1 - \theta_0}{\sigma^2} \left( X_i - \frac{\theta_0 + \theta_1}{2} \right)$. - Drift and variance: $\mathbb{E}_\theta[Z_i] = \frac{\theta_1 - \theta_0}{\sigma^2} \left( \theta - \frac{\theta_0 + \theta_1}{2} \right)$, $\text{Var}_\theta(Z_i) = \frac{(\theta_1 - \theta_0)^2}{\sigma^2}$. Zero-drift midpoint: $\theta^* = \frac{\theta_0 + \theta_1}{2}$. - Explicit root $h(\theta)$: Since $Z_i \sim \mathcal{N}(\mu_Z, \sigma_Z^2)$, its MGF is $\exp(t \mu_Z + \frac{1}{2} t^2 \sigma_Z^2) = 1$, which gives the exact closed-form root $h(\theta) = -\frac{2\mu_Z}{\sigma_Z^2} = \frac{\theta_0 + \theta_1 - 2\theta}{\theta_1 - \theta_0}$. 2. **Bernoulli Proportion (Binomial Sequential Sampling)**: $X_i \stackrel{\text{i.i.d.}}{\sim} \text{Bernoulli}(p)$ for testing $H_0 : p = p_0$ vs. $H_1 : p = p_1$ ($0 < p_0 < p_1 < 1$). - Log-likelihood increment: $Z_i = X_i \ln\left(\frac{p_1(1 - p_0)}{p_0(1 - p_1)}\right) + \ln\left(\frac{1 - p_1}{1 - p_0}\right)$. - Expected increment: $\mathbb{E}_p[Z_i] = p \ln\left(\frac{p_1}{p_0}\right) + (1 - p) \ln\left(\frac{1 - p_1}{1 - p_0}\right)$. - Characteristic root equation: $M_p(h) = p \left(\frac{p_1}{p_0}\right)^h + (1 - p) \left(\frac{1 - p_1}{1 - p_0}\right)^h = 1$, which yields the explicit parametric expression for $p$ as a function of $h$: $$p(h) = \frac{1 - \left(\frac{1 - p_1}{1 - p_0}\right)^h}{\left(\frac{p_1}{p_0}\right)^h - \left(\frac{1 - p_1}{1 - p_0}\right)^h}$$ 3. **Poisson Rate**: $X_i \stackrel{\text{i.i.d.}}{\sim} \text{Poisson}(\lambda)$ for testing $H_0 : \lambda = \lambda_0$ vs. $H_1 : \lambda = \lambda_1$ ($\lambda_1 > \lambda_0$). - Log-likelihood increment: $Z_i = X_i \ln\left(\frac{\lambda_1}{\lambda_0}\right) - (\lambda_1 - \lambda_0)$. - Expected increment: $\mathbb{E}_\lambda[Z_i] = \lambda \ln\left(\frac{\lambda_1}{\lambda_0}\right) - (\lambda_1 - \lambda_0)$, with zero-drift point $\lambda^* = \frac{\lambda_1 - \lambda_0}{\ln(\lambda_1 / \lambda_0)}$. - Parametric root relation: $\lambda(h) = \frac{h(\lambda_1 - \lambda_0)}{(\lambda_1 / \lambda_0)^h - 1}$. 4. **Exponential Scale/Rate**: $X_i \stackrel{\text{i.i.d.}}{\sim} \text{Exponential}(\theta)$ with density $f(x; \theta) = \theta e^{-\theta x}$ ($x > 0$) for testing $H_0 : \theta = \theta_0$ vs. $H_1 : \theta = \theta_1$ ($\theta_1 > \theta_0$). - Log-likelihood increment: $Z_i = \ln\left(\frac{\theta_1}{\theta_0}\right) - (\theta_1 - \theta_0) X_i$. - Expected increment: $\mathbb{E}_\theta[Z_i] = \ln\left(\frac{\theta_1}{\theta_0}\right) - \frac{\theta_1 - \theta_0}{\theta}$.

Why did Abraham Wald select constant horizontal boundaries $(b, a)$ for the cumulative log-likelihood ratio $S_m$ rather than curved or time-varying boundaries $(b_m, a_m)$? In 1948, Abraham Wald and Jacob Wolfowitz proved one of the deepest theorems in mathematical statistics: among all sequential and fixed-sample tests whose error probabilities do not exceed those of the SPRT, the SPRT simultaneously minimizes the expected sample size under both $H_0$ and $H_1$!

Theorem 5.5 (Wald-Wolfowitz Optimality Theorem for the SPRT)

Let $\mathcal{T}^*$ be a Sequential Probability Ratio Test with stopping time $N^*$ and terminal decision $\delta^*$ having error probabilities $\alpha(\mathcal{T}^*) = \alpha$ and $\beta(\mathcal{T}^*) = \beta$. Let $\mathcal{T} = (N, \delta)$ be any other sequential or fixed-sample test such that:

Then the expected sample size of $\mathcal{T}^*$ is simultaneous smaller than or equal to that of $\mathcal{T}$ under both hypotheses:

Sketch of Proof via Bayesian Sequential Decision Theory and Kullback-Leibler Divergence:

Consider a Bayesian sequential decision problem where $\theta \in \{\theta_0, \theta_1\}$ is assigned prior probability $\pi = \mathbb{P}(\theta = \theta_1) \in (0, 1)$, let $c > 0$ be the cost per observation, and let $w_0, w_1 > 0$ be the terminal losses for Type I and Type II errors. The total Bayes risk of any test $\mathcal{T} = (N, \delta)$ is:

By backward induction and Bellman's optimality equation for Markov decision processes, since the posterior probability $\pi_m = \mathbb{P}(\theta = \theta_1 \mid X_1, \dots, X_m)$ is a monotone function of the likelihood ratio $\Lambda_m$ via Bayes' rule:

and the future observation dynamics are time-homogeneous, the optimal Bayes continuation region in terms of $\pi_m$ is a time-invariant open interval $(\pi_L, \pi_U)$, which transforms via Bayes' rule into a time-invariant interval $B < \Lambda_m < A$, i.e., an SPRT! By varying the prior $\pi$ and loss weights $(w_0, w_1)$ to match the given boundaries $(B, A)$, if any competing test $\mathcal{T}$ had $\alpha(\mathcal{T}) \le \alpha(\mathcal{T}^*)$, $\beta(\mathcal{T}) \le \beta(\mathcal{T}^*)$, and strictly smaller $\mathbb{E}_{\theta_0}[N] < \mathbb{E}_{\theta_0}[N^*]$ (or $\mathbb{E}_{\theta_1}[N] < \mathbb{E}_{\theta_1}[N^*]$), it would achieve a strictly smaller Bayes risk than the Bayes-optimal SPRT $\mathcal{T}^*$, which is a contradiction.

We can also derive a universal Information-Theoretic Lower Bound on $\mathbb{E}_{\theta_0}[N]$ and $\mathbb{E}_{\theta_1}[N]$ directly from Wald's Equation and the data-processing inequality for Kullback-Leibler divergence. Note that $\mathbb{E}_{\theta_0}[Z_1] = -D_{\text{KL}}(f_0 \parallel f_1)$. Applying Jensen's inequality conditionally on the terminal decision $\delta_N \in \{0, 1\}$ to $\mathbb{E}_{\theta_0}[S_N] = \mathbb{E}_{\theta_0}[\ln \Lambda_N]$ yields:

Thus, for any test $\mathcal{T} = (N, \delta)$ with error probabilities $(\alpha, \beta)$:

Comparing the numerator of this universal lower bound with Wald's ASN formula at $\theta_0$ and $\theta_1$ using $\ln A \approx \ln\left(\frac{1-\beta}{\alpha}\right)$ and $\ln B \approx \ln\left(\frac{\beta}{1-\alpha}\right)$, we see that Wald's SPRT attains the exact information-theoretic lower bound up to the negligible boundary overshoot!

In clinical trials and industrial applications, one practical concern with the open-ended SPRT is that if the true parameter happens to lie near the indifference point $\theta^*$, the sample size $N$, though finite with probability one, has a positively skewed distribution and may occasionally exceed the fixed-sample size $n_{\text{fixed}}$. Furthermore, budgeting and hospital logistics require a hard upper bound $N_{\max}$ on the number of patients enrolled. This motivates Truncated Sequential Designs and Group Sequential Designs:

• Wald's Truncated SPRT: If the SPRT has not terminated by a pre-specified maximum stage $N_{\max}$, sampling is forcibly stopped at $N_{\max}$, and $H_0$ is accepted if $S_{N_{\max}} \le 0$ and rejected if $S_{N_{\max}} > 0$.
• Group Sequential Clinical Trials (Pocock, O'Brien-Fleming, and Lan-DeMets $\alpha$-Spending Boundaries): Rather than monitoring after every single patient, data are analyzed at $K$ interim looks (e.g., $K = 4$ or $5$ stages of $m$ patients each). Pocock (1977) proposed constant standardized $Z$-thresholds across all looks, whereas O'Brien and Fleming (1979) proposed conservative early thresholds $c_k = c_{\text{OBF}} \sqrt{K/k}$ that make early stopping very stringent unless the treatment effect is dramatic, while preserving a final critical value $c_K$ nearly identical to the fixed-sample test.

---