Methodological Validity: Confounding, Selection Bias, Information Bias & Effect Modification

Scientific validity refers to the degree to which an empirical study reaches true conclusions about the phenomenon under investigation. Epidemiologists partition validity into two primary domains: 1. **Internal Validity**: The degree to which the observed association between exposure and disease is free from systematic error within the specific study population. An internally valid study accurately estimates the true causal effect that operated among the enrolled participants. 2. **External Validity (Generalizability / Transportability)**: The extent to which causal findings established within the study population can be legitimately applied or generalized to external target populations that differ in demographic, geographic, or temporal characteristics. Crucially, **internal validity is a strict logical prerequisite for external validity**: a study that is internally flawed by confounding or misclassification cannot be generalizable to anyone. ### Taxonomy of Epidemiologic Errors Total error in an observational estimate decomposes into two fundamentally distinct components: $$\text{Total Error} = \text{Random Error} + \text{Systematic Error (Bias)}$$ Selection bias occurs when the procedures used to enroll participants or retain them over time create an analytic sample in which the relationship between exposure and disease differs from that in the target population (such as Berkson's bias in hospital studies or healthy worker survivor bias). Information bias arises from imperfect measurement or categorization of exposure, covariate, or outcome variables.

Confounding is the confusion or mixing of the true causal effect of the exposure with the extraneous effects of other factors that influence the outcome.

The Classical Epidemiologic Definition

Historically, a variable $Z$ was defined as a confounder of the association between exposure $E$ and disease $D$ if it satisfied three simultaneous criteria:

1. $Z$ must be causally or statistically associated with the exposure $E$ in the source population.
2. $Z$ must be an independent risk factor for the disease $D$ among unexposed individuals.
3. $Z$ must not be an intermediate step on the biological causal pathway between $E$ and $D$ ($E \to Z \to D$).

While useful, this classical triad is neither strictly necessary nor sufficient in complex multivariable settings.

Modern Counterfactual Definition and Exchangeability

Modern epidemiologic theory formalizes confounding through the Neyman-Rubin Potential Outcomes framework. Let $Y_i(1)$ denote the counterfactual health outcome that individual $i$ would experience if exposed ($E = 1$), and let $Y_i(0)$ denote the counterfactual outcome that the same individual would experience if unexposed ($E = 0$).

The true Average Causal Effect ($ACE$) in the population is:

In real life, we observe only the realized outcome $Y = E \cdot Y(1) + (1 - E) \cdot Y(0)$. We can observe $\mathbb{E}[Y \mid E = 1] = \mathbb{E}[Y(1) \mid E = 1]$ and $\mathbb{E}[Y \mid E = 0] = \mathbb{E}[Y(0) \mid E = 0]$, but we cannot observe the counterfactual unexposed risk among the exposed ($\mathbb{E}[Y(0) \mid E = 1]$).

Confounding is formally defined as the failure of exchangeability:

If the unexposed group does not accurately represent what would have happened to the exposed group had they remained unexposed, the groups are non-exchangeable, and the crude observational comparison is confounded.

Causal Directed Acyclic Graphs (DAGs) and Pearl's Backdoor Criterion

Judea Pearl formalized causal diagrams using Directed Acyclic Graphs (DAGs). In a DAG, nodes represent random variables, and directed arrows ($\to$) represent direct causal mechanisms.

• A path is an unbroken sequence of adjacent edges regardless of arrow direction.
• A causal path from $E$ to $D$ consists exclusively of forward-pointing arrows: $E \to \dots \to D$.
• A backdoor path is any non-causal path connecting $E$ and $D$ that begins with an arrow pointing into $E$ ($E \leftarrow \dots \to D$). Backdoor paths transmit spurious, non-causal statistical associations.

A path is blocked (d-separated) if it contains:

1. A chain ($A \to B \to C$) or fork ($A \leftarrow B \to C$) where the intermediate variable $B$ is conditioned upon (controlled for in regression or stratification).
2. A collider ($A \to C \leftarrow B$), where two arrows collide into node $C$. Crucially, a collider naturally blocks the path without conditioning!

Pearl's Backdoor Criterion: A set of covariates $\mathbf{Z}$ satisfies the Backdoor Criterion relative to $(E, D)$ if:

1. No node in $\mathbf{Z}$ is a descendant of $E$.
2. $\mathbf{Z}$ blocks every backdoor path between $E$ and $D$.

Collider Stratification Bias and M-Bias

If an investigator conditions on a collider $C$ (or a descendant of a collider), the path becomes unblocked, inducing a spurious, non-causal association between $A$ and $B$. This phenomenon is termed collider stratification bias.

A celebrated manifestation is M-Bias, illustrated by the DAG:

Here, $U_1$ and $U_2$ are unobserved variables, and $M$ is a pre-treatment covariate that is caused by both $U_1$ and $U_2$. In the unadjusted data, the backdoor path from $E$ to $D$ is completely blocked because $M$ is an unconditioned collider ($U_1 \to M \leftarrow U_2$). The crude association between $E$ and $D$ is unconfounded! However, if a naive analyst adjusts for $M$ in a multivariable regression model, conditioning on the collider $M$ opens the backdoor path, artificially creating confounding where none previously existed!

**Stratification** is a classical, non-parametric analytical technique for controlling confounding. The total study population is partitioned into $K$ mutually exclusive and exhaustive strata defined by the categories of the confounding variable $Z$ (e.g., age groups, smoking categories). Within each stratum $k \in \{1, 2, \dots, K\}$, the confounding variable $Z$ is held constant, eliminating its ability to confound the exposure-outcome association. Within stratum $k$, data form a $2 \times 2$ table: ### The Mantel-Haenszel Pooled Odds Ratio Estimator William G. Cochran (1954) and Nathan Mantel and William Haenszel (1959) developed the **Mantel-Haenszel Pooled Odds Ratio** ($OR_{MH}$) as an optimal weighted average across the $K$ strata: $$OR_{MH} = \frac{\sum_{k=1}^K \frac{a_k d_k}{N_k}}{\sum_{k=1}^K \frac{b_k c_k}{N_k}}$$ Notice that $a_k d_k / (b_k c_k)$ is the stratum-specific odds ratio $OR_k$. The Mantel-Haenszel estimator assigns each stratum a weight proportional to: $$w_k = \frac{b_k c_k}{N_k}$$ which is nearly inversely proportional to the stratum-specific variance under the hypergeometric null distribution, guaranteeing high statistical efficiency. ### Variance of ln(OR_MH): The Robins-Breslow-Greenland Formula James Robins, Norman Breslow, and Sander Greenland (1986) derived the definitive dually consistent variance estimator for the natural logarithm of $OR_{MH}$, valid both when the number of strata $K$ is small and strata are large (large-stratum limiting model) and when $K$ is large with sparse cell counts (sparse-data limiting model, such as 1:1 matched pairs): $$\text{Var}(\ln OR_{MH}) = \frac{\sum_{k=1}^K P_k R_k}{2 R_+^2} + \frac{\sum_{k=1}^K (P_k S_k + Q_k R_k)}{2 R_+ S_+} + \frac{\sum_{k=1}^K Q_k S_k}{2 S_+^2}$$ where: - $R_k = \frac{a_k d_k}{N_k}, \quad S_k = \frac{b_k c_k}{N_k}$ - $R_+ = \sum_{k=1}^K R_k, \quad S_+ = \sum_{k=1}^K S_k \implies OR_{MH} = \frac{R_+}{S_+}$ - $P_k = \frac{a_k + d_k}{N_k}, \quad Q_k = \frac{b_k + c_k}{N_k}$ A 95% confidence interval is evaluated as $OR_{MH} \cdot \exp(\pm 1.96 \sqrt{\text{Var}(\ln OR_{MH})})$. ### The Mantel-Haenszel Summary Risk Ratio and Risk Difference Analogously, the Mantel-Haenszel pooled Risk Ratio and Risk Difference across $K$ cohort strata are: $$RR_{MH} = \frac{\sum_{k=1}^K \frac{a_k n_{0k}}{N_k}}{\sum_{k=1}^K \frac{c_k n_{1k}}{N_k}}$$ $$RD_{MH} = \frac{\sum_{k=1}^K \frac{n_{1k} n_{0k}}{N_k} \left( \frac{a_k}{n_{1k}} - \frac{c_k}{n_{0k}} \right)}{\sum_{k=1}^K \frac{n_{1k} n_{0k}}{N_k}}$$ ### The Cochran-Mantel-Haenszel Chi-Square Test To test the global null hypothesis that exposure is conditionally independent of disease across all strata ($H_0: OR_1 = OR_2 = \dots = OR_K = 1$), the **CMH test statistic** compares observed exposed cases $a_k$ to their hypergeometric expectation under fixed marginal totals: $$\mathbb{E}[a_k \mid H_0] = \frac{n_{1k} m_{1k}}{N_k}$$ $$\text{Var}(a_k \mid H_0) = \frac{n_{1k} n_{0k} m_{1k} m_{0k}}{N_k^2 (N_k - 1)}$$ Summing over all $K$ strata yields the 1-degree-of-freedom test statistic: $$\chi^2_{CMH} = \frac{\left( \left| \sum_{k=1}^K a_k - \sum_{k=1}^K \mathbb{E}[a_k \mid H_0] \right| - 0.5 \right)^2}{\sum_{k=1}^K \text{Var}(a_k \mid H_0)}$$

While confounding is a methodological artifact resulting from an uneven distribution of extraneous risk factors that the investigator must eliminate, effect measure modification (also termed heterogeneity of effect or statistical interaction) is a genuine biological reality that describes how the effect of an exposure varies across subpopulations.

Dependence on the Mathematical Scale

Crucially, effect measure modification depends entirely on the mathematical scale of measurement:

1. Additive Interaction: Occurs when the Risk Difference ($RD$) varies across levels of a third factor $Z$:

2. Multiplicative Interaction: Occurs when the Risk Ratio ($RR$) or Odds Ratio ($OR$) varies across levels of $Z$:

It is a mathematical theorem of epidemiology that unless the effect of exposure is zero, interaction cannot be simultaneously absent on both the additive and multiplicative scales. If the relative risk is constant across age strata ($RR = 2.0$), the absolute risk difference must inevitably increase with age because baseline risk $R_0$ increases with age ($RD = R_0(RR - 1)$).

Evaluating Biological Interaction: RERI and the Synergy Index

Kenneth Rothman (1976) demonstrated that biological interaction (mechanistic synergy in sufficient-component cause models) corresponds to departure from additivity of risks, rather than multiplicativity. When analyzing case-control data where only odds ratios (relative risks) are available, epidemiologists evaluate departures from additive risk differences using the Relative Excess Risk due to Interaction ($RERI$), also known as the Interaction Contrast Ratio ($ICR$):

Let $RR_{11}$ denote the relative risk with both exposures present, $RR_{10}$ the relative risk with exposure 1 alone, $RR_{01}$ the relative risk with exposure 2 alone, and $RR_{00} = 1.0$ the unexposed reference. Under an additive null model:

• $RERI = 0$: Perfect additivity of effects; no biological interaction.
• $RERI > 0$: Positive interaction (super-additive synergy); the combined effect of both exposures exceeds the sum of their individual excess risks.
• $RERI < 0$: Negative interaction (sub-additive antagonism).

The Synergy Index ($S$) measures the ratio of the observed combined excess risk to the sum of the individual excess risks:

If $S > 1.0$, super-additive synergy is confirmed. A classic historical example is the interaction between asbestos exposure and cigarette smoking in lung carcinogenesis: while smoking alone produces $RR_{10} \approx 10.0$ and asbestos alone produces $RR_{01} \approx 5.0$, the combined exposure produces $RR_{11} \approx 50.0$, yielding $S = (50 - 1) / [(10 - 1) + (5 - 1)] = 49 / 13 \approx 3.77$.

Before pooling stratum-specific odds ratios into a single summary Mantel-Haenszel estimate, the investigator must verify that the odds ratio is homogeneous across strata ($H_0: OR_1 = OR_2 = \dots = OR_K = OR$). If substantial effect modification is present, presenting a single summary pooled odds ratio is scientifically misleading; the stratum-specific effect estimates should be reported separately.

The Breslow-Day Test of Homogeneity

Norman Breslow and Nicholas Day (1980) formulated an asymptotic test statistic comparing the observed count $a_k$ in each stratum to the fitted count $A_k^*$ that would be expected if the common odds ratio equaled the Mantel-Haenszel estimate $OR_{MH}$. The fitted count $A_k^*$ is the positive root of the quadratic equation:

The Breslow-Day Test Statistic is:

where $\text{Var}(a_k \mid A_k^*) = \left( \frac{1}{A_k^*} + \frac{1}{n_{1k} - A_k^*} + \frac{1}{m_{1k} - A_k^*} + \frac{1}{n_{0k} - m_{1k} + A_k^*} \right)^{-1}$. Under the null hypothesis of homogeneity, $\chi^2_{BD}$ asymptotically follows a chi-square distribution with $K - 1$ degrees of freedom.

Tarone's Adjustment

Robert Tarone (1985) proved that because $OR_{MH}$ is not the efficient maximum likelihood estimator of the common odds ratio, the Breslow-Day test can fail to attain its nominal asymptotic chi-square distribution. Tarone introduced a score adjustment:

which guarantees exact asymptotic validity with $K - 1$ degrees of freedom.

Information bias occurs when data regarding exposure status, covariate levels, or disease outcomes are systematically mismeasured, misclassified, or incorrectly recorded. ### Non-Differential vs Differential Misclassification In categorical epidemiology, measurement error is designated as **misclassification**. We define: - **Sensitivity of Classification** ($Se$): The probability that a truly positive individual is correctly categorized ($P(E_{\text{obs}}^+ \mid E_{\text{true}}^+)$). - **Specificity of Classification** ($Sp$): The probability that a truly negative individual is correctly categorized ($P(E_{\text{obs}}^- \mid E_{\text{true}}^-)$). We categorize misclassification along two structural axes: ### Quantitative Bias Analysis (QBA) To reconstruct the true contingency table from an observed, misclassified table $(A, B, C, D)$, epidemiologists employ matrix inversion. Let $A$ and $B$ denote the observed exposed and unexposed counts. If sensitivity $Se$ and specificity $Sp$ of exposure measurement are known from an internal validation sub-study: $$A = a_{\text{true}} \cdot Se + b_{\text{true}} \cdot (1 - Sp)$$ Solving for true counts yields: $$a_{\text{true}} = \frac{A - (a_{\text{true}} + b_{\text{true}})(1 - Sp)}{Se + Sp - 1} = \frac{A - N_{\text{row}}(1 - Sp)}{Se + Sp - 1}$$ Quantitative bias analysis enables investigators to evaluate the robustness of published causal claims against realistic levels of measurement error.

Beyond the classical Backdoor Criterion, graphical causal inference provides alternative topological structures for identifying causal effects even in the presence of unmeasured confounding:

Pearl's Frontdoor Criterion

Suppose an unmeasured confounder $U$ severely confounds the direct link between exposure $E$ and outcome $D$ ($E \leftarrow U \to D$), such that no backdoor adjustment can block the path. If there exists an intermediate variable $M$ on the causal pathway ($E \to M \to D$) that satisfies the Frontdoor Criterion:

1. $E$ intercepts all directed causal paths from $E$ to $D$ via $M$.
2. There are no unblocked backdoor paths from $E$ to $M$.
3. All backdoor paths from $M$ to $D$ are blocked by conditioning on $E$.

Then the average causal effect of $E$ on $D$ is completely identified through the frontdoor adjustment formula:

This celebrated theorem demonstrates that causal effects can be identified even when an exposure and outcome share intractable unmeasured confounding!

Instrumental Variables and Mendelian Randomization

An Instrumental Variable ($Z$) is an exogenous variable that satisfies three strict core assumptions:

1. Relevance: $Z$ has a causal effect on exposure $E$ ($Z \to E$).

2. Exclusion Restriction: $Z$ affects outcome $D$ exclusively through its effect on $E$, with no direct causal pathway ($Z \to D$ is absent).

3. Independence (Unconfoundedness): $Z$ shares no common causes with $D$ ($Z \leftarrow U \to D$ is absent).

In genetic epidemiology, Mendelian Randomization exploits inherited genetic variants (e.g., single nucleotide polymorphisms, SNPs) as natural instrumental variables. Because alleles are randomly assorted during meiosis at gametogenesis, genetic inheritance acts as a lifelong natural randomized controlled trial, permitting unconfounded estimation of modifiable behavioral or biochemical risk factors (such as circulating LDL cholesterol or uric acid) on chronic cardiovascular endpoints via two-stage least squares estimators:

Information bias occurs when study variables (exposure, disease, or confounders) are measured, classified, or recorded with systematic error. The mathematical impact of exposure misclassification depends fundamentally on whether classification errors are conditional on disease status.

Let $A^* \in \{0, 1\}$ denote the imperfectly measured (classified) binary exposure, and let $A \in \{0, 1\}$ denote true exposure status.

• Sensitivity of exposure classification: $Se_A = P(A^* = 1 \mid A = 1)$.
• Specificity of exposure classification: $Sp_A = P(A^* = 0 \mid A = 0)$.

1. Non-Differential Exposure Misclassification

Misclassification of exposure is non-differential with respect to disease outcome $Y$ if sensitivity and specificity do not vary according to disease status:

Under non-differential binary exposure misclassification with independent errors (where $Se_A + Sp_A > 1$):

• The observed risk ratio $RR^*$ and observed odds ratio $OR^*$ are strictly biased toward the null value of 1.0:

Non-differential misclassification dilutes observed effect sizes, obscuring real epidemiological signals.

2. The Rogan-Gladen Truth Correction Formula

To recover the true prevalence of exposure $P(A = 1)$ from observed classified prevalence $P(A^* = 1)$:

Solving explicitly for true exposure prevalence produces the Rogan-Gladen Formula:

This matrix inversion formula allows epidemiologists to perform quantitative bias analysis (QBA) and reconstruct true measures of association from validation sub-studies.