Process Capability Analysis, Tolerance Synthesis & Measurement Systems Evaluation

A foundational distinction in industrial quality engineering separates the statistical voice of the process from the engineering voice of the customer:

• Control Limits ($LCL, UCL$): Derived exclusively from the inherent variability of the manufacturing process ($\mu \pm 3\sigma/\sqrt{n}$). They reflect what the process is actually achieving when operating stably under common causes.
• Specification Limits ($LSL, USL$): Established by product designers, systems architects, and customer functional requirements. They reflect what the manufactured part must achieve to assemble properly, withstand operational stress, and prevent operational failure.

There is no necessary mathematical connection between control limits and specification tolerances. A process can be in perfect statistical control (highly stable and predictable) yet produce $100\%$ nonconforming scrap if its natural variation exceeds the allowable tolerance spread or if its operating mean is severely off-center. Conversely, an out-of-control process might temporarily produce parts that happen to fall within specifications, but its unpredictability poses severe long-term risk. Process Capability Analysis quantitatively bridges this gap, evaluating whether the natural distribution of a stable process fits comfortably within the engineering tolerance boundaries.

Assume a continuous quality characteristic $X$ is generated by an in-control manufacturing process following a normal distribution $X \sim \mathcal{N}(\mu, \sigma^2)$, with Lower Specification Limit ($LSL$) and Upper Specification Limit ($USL$).

The total allowable engineering tolerance width is:

The natural process capability spread is defined by Shewhart's $6\sigma$ envelope (from $\mu - 3\sigma$ to $\mu + 3\sigma$), encompassing $99.73\%$ of output under normality. The Potential Process Capability Index ($C_p$) is defined as the ratio of allowable tolerance to natural process spread:

Interpretation of $C_p$

• $C_p < 1.0$ (Incapable Process): The natural process spread is wider than the tolerance band ($6\sigma > USL - LSL$). Even if the process is centered perfectly at the nominal midpoint $M = (USL + LSL)/2$, a substantial fraction of parts will inevitably breach specification limits, generating defective product.
• $C_p = 1.0$ (Barely Capable Process): The $6\sigma$ spread exactly equals the tolerance bandwidth. When perfectly centered, the expected nonconforming fraction is $0.27\%$ ($2,700$ parts per million, PPM). Any slight disturbance in process mean immediately produces scrap.
• $C_p = 1.33$ (Capable Process, 4-Sigma Quality): The tolerance band spans $8\sigma$. When centered, the defect rate is $63\text{ PPM}$. This is the traditional industrial benchmark for established manufacturing processes.
• $C_p \ge 1.67$ (Highly Capable, 5-Sigma Quality): The tolerance band spans $10\sigma$, with a centered defect rate of $0.57\text{ PPM}$. Required for critical safety components in aerospace and medical technology.
• $C_p \ge 2.0$ (World-Class, Six Sigma Quality): The tolerance band spans $12\sigma$. Even under a sustained $1.5\sigma$ mean drift, the process produces no more than $3.4\text{ PPM}$ nonconforming output.

Notice that $C_p$ evaluates only the process spread relative to tolerance; it provides zero information regarding whether the process is actually centered between $LSL$ and $USL$.

To penalize process off-centering, quality engineering defines unilateral capability ratios for the upper and lower specification limits:

The Actual Process Capability Index ($C_{pk}$) is the minimum of these two one-sided ratios:

Algebraically, $C_{pk}$ can be related directly to $C_p$ via the centering parameter $k$:

where $k \in [0, 1]$ represents the fractional deviation of the mean from nominal midpoint. Then:

Key Properties of $C_{pk}$

1. If the process is perfectly centered ($\mu = M$), then $k = 0$, and $C_{pk} = C_p$.
2. As the process mean shifts away from center, $k$ increases, causing $C_{pk}$ to decrease monotonically while $C_p$ remains constant.
3. If the process mean shifts exactly onto one of the specification limits ($\mu = USL$ or $\mu = LSL$), $C_{pk} = 0$, indicating that $50\%$ of production is defective.
4. If the process mean shifts outside the specification limits, $C_{pk} < 0$.
5. The actual fraction of nonconforming product produced by the process is given by:

A common upper bound on the nonconforming fraction is $p \le 2\Phi(-3 C_{pk})$.

Genichi Taguchi challenged the traditional "goal-post" mentality of specification limits. The goal-post philosophy assumes that a product dimension falling just inside specification ($USL - \epsilon$) incurs zero loss, whereas a dimension falling just outside ($USL + \epsilon$) incurs immediate catastrophic loss. Taguchi demonstrated that customer dissatisfaction and economic cost accumulate continuously as a quadratic function of deviation from the design target $T$:

where $k_T$ is a financial loss coefficient. The expected quality loss per unit manufactured is:

Taguchi introduced the $C_{pm}$ Index (also known as the Taguchi Capability Index), which incorporates the mean square deviation from the design target $T$:

Notice that $\tau$ reflects both process variance $\sigma^2$ and off-target bias $(\mu - T)^2$. We can rewrite $C_{pm}$ in terms of $C_p$:

where $\xi = (\mu - T)/\sigma$ is the standardized target discrepancy. If the process mean is centered exactly on target ($\mu = T$), then $C_{pm} = C_p$. If the process mean drifts away from target, $C_{pm}$ drops rapidly, penalizing the departure even if the process remains comfortably within traditional specification tolerances.

In empirical industrial practice, true parameters $\mu$ and $\sigma$ are unknown and must be estimated from a random sample of $N$ parts, yielding sample estimators:

Because $\widehat{C}_p$ and $\widehat{C}_{pk}$ are sample statistics, they exhibit sampling variability. Making operational decisions based solely on point estimates without confidence intervals introduces substantial commercial risk.

Sampling Distribution of $\widehat{C}_p$

Recall that for a normal sample of size $N$, $(N - 1)S^2 / \sigma^2 \sim \chi^2_{N-1}$. Therefore:

Thus, $(N - 1)(\widehat{C}_p / C_p)^2$ follows an inverted chi-square distribution. A $(1 - \alpha)100\%$ confidence interval for the true potential capability $C_p$ is:

Asymptotic Distribution and Confidence Interval for $\widehat{C}_{pk}$

Zhang, Stenback, and Wardrop (1990) derived the large-sample asymptotic variance of $\widehat{C}_{pk}$:

An approximate $(1 - \alpha)100\%$ confidence interval for $C_{pk}$ is:

This formula reveals why small sample sizes (such as $N = 30$) yield excessively wide confidence intervals, frequently causing engineers to overestimate true process capability.

Before process capability can be meaningfully assessed, the measurement system itself must be validated. If the calipers, coordinate measuring machines (CMM), optical comparators, or sensors exhibit substantial measurement noise, the observed process variance will be artificially inflated:

The variance of the measurement system $\sigma^2_{\text{measurement}}$ (commonly termed Gage R&R, $\sigma^2_{GRR}$) is partitioned into two orthogonal components:

1. Repeatability ($\sigma^2_{\text{repeatability}}$, Equipment Variation, $EV$): The variability observed when one operator measures the exact same part multiple times using the identical measurement instrument.

2. Reproducibility ($\sigma^2_{\text{reproducibility}}$, Appraiser Variation, $AV$): The variability observed when different operators measure the exact same part using the identical measurement instrument.

ANOVA Method for Gage R&R

In a standard crossed Gage R&R study, $p$ parts are selected at random from the production process, $o$ operators are assigned, and each operator measures every part $r$ times (replicates). The total number of observations is $N = p \cdot o \cdot r$.

The two-factor crossed random effects model is:

where $P_i \sim \mathcal{N}(0, \sigma^2_P)$ is the random part effect, $O_j \sim \mathcal{N}(0, \sigma^2_O)$ is the random operator effect, $(PO)_{ij} \sim \mathcal{N}(0, \sigma^2_{PO})$ is the part-by-operator interaction, and $\varepsilon_{ijk} \sim \mathcal{N}(0, \sigma^2_{\varepsilon})$ is the experimental measurement error.

From the two-way ANOVA table:

• Repeatability Variance: $\widehat{\sigma}^2_{\text{repeatability}} = MS_E$
• Part-by-Operator Interaction: $\widehat{\sigma}^2_{PO} = \frac{MS_{PO} - MS_E}{r}$
• Operator Reproducibility: $\widehat{\sigma}^2_O = \frac{MS_O - MS_{PO}}{p \cdot r}$
• Gage Reproducibility: $\widehat{\sigma}^2_{\text{reproducibility}} = \widehat{\sigma}^2_O + \widehat{\sigma}^2_{PO}$
• Total Measurement System Variance: $\widehat{\sigma}^2_{GRR} = \widehat{\sigma}^2_{\text{repeatability}} + \widehat{\sigma}^2_{\text{reproducibility}}$
• Part-to-Part Variance: $\widehat{\sigma}^2_{\text{part}} = \frac{MS_P - MS_{PO}}{o \cdot r}$

Measurement System Evaluation Metrics

The Automotive Industry Action Group (AIAG) defines three primary metrics:

1. Precision-to-Tolerance Ratio ($P/T$):

2. Percentage of Total Process Variation ($\%GRR$):

3. Number of Distinct Categories ($ndc$):

Acceptance Criteria

• $P/T$ or $\%GRR < 10\%$: Measurement system is excellent and universally acceptable.
• $10\% \le P/T \le 30\%$: Measurement system is conditionally acceptable depending on application criticality and replacement cost.
• $P/T > 30\%$: Measurement system is unacceptable; calibration, fixturing, or instrument replacement is mandatory.
• $ndc \ge 5$: The measurement system possesses adequate statistical resolution to discriminate between parts.

The standard formulations of $C_p$ and $C_{pk}$ assume that the underlying process follows a Gaussian distribution. When process data exhibit significant skewness or kurtosis (for instance, flatness, surface roughness, or cycle times, which are strictly non-negative and inherently right-skewed), applying standard Gaussian formulas produces catastrophic errors: the estimated defect rate can differ from reality by orders of magnitude.

To analyze non-normal processes without distorting original units, John A. Clements (1989) formulated a distribution-free capability methodology based on the Pearson System of Frequency Curves. In Clements' method, the natural process spread is defined not by $6\sigma$, but by the difference between the 99.865th percentile and the 0.135th percentile of the true fitted distribution, which matches the coverage of a Gaussian $6\sigma$ span ($99.73\%$):

The Clements Potential Capability Index is defined as:

To account for centering, Clements defined upper and lower capability ratios referenced to the median ($X_{0.50}$):

Clements published extensive lookup tables parameterized by the sample skewness $\sqrt{b_1}$ and kurtosis $b_2$, allowing practitioners to determine standardized percentile multipliers without complex numerical integration. Alternatively, modern practitioners apply the Box-Cox or Johnson transformations to normalize the empirical data before evaluating $C_{pk}$ on the transformed scale.

Modern precision assemblies frequently require simultaneous compliance across $p$ correlated geometric or mechanical characteristics $\mathbf{X} = (X_1, X_2, \dots, X_p)^T \sim \mathcal{N}_p(\boldsymbol{\mu}, \boldsymbol{\Sigma})$. Evaluating unilateral capability indices $C_{pk, j}$ for each dimension independently commits the independence fallacy: even if each individual dimension satisfies $C_{pk} \ge 1.33$, the joint probability that all $p$ dimensions simultaneously fall within their specification region may be unacceptably low due to covariance.

Let the allowable engineering tolerance region be a $p$-dimensional hyper-rectangle:

Taam, Subbaiah, and Liddy (1993) introduced the Multivariate Capability Index ($MC_p$), defined as the ratio of the volume of the modified tolerance region to the volume of the $99.73\%$ process tolerance ellipse:

To incorporate centering, Taam et al. formulated the $MC_{pm}$ index:

where $(\boldsymbol{\mu} - \mathbf{T})^T \boldsymbol{\Sigma}^{-1} (\boldsymbol{\mu} - \mathbf{T})$ is the squared Mahalanobis distance between the operating mean vector $\boldsymbol{\mu}$ and the engineering target vector $\mathbf{T}$. If the process drifts off-target in multivariate space, $MC_{pm}$ declines rapidly, warning engineers of compound assembly interference.

In addition to repeatability and reproducibility, a complete measurement system evaluation requires evaluating two fundamental properties: Bias and Linearity.

Measurement Bias

Bias is the difference between the observed average of measurements and a certified reference standard master value $X_{\text{master}}$:

To statistically verify whether bias is significant, an operator measures the reference standard $m$ times ($m \ge 15$). A one-sample Student $t$-test is performed:

Under the null hypothesis $H_0: \text{Bias} = 0$, $t_{\text{stat}}$ follows a $t$-distribution with $m - 1$ degrees of freedom. If $p < 0.05$, the measurement gage has statistically significant bias, requiring physical recalibration or electronic zero-offset adjustment.

Measurement Linearity

A measurement device may exhibit zero bias at the midpoint of its operating range, yet display negative bias at the lower end and positive bias at the upper end. Linearity assesses whether the measurement bias remains constant across the entire intended operating range of the instrument.

To perform a linearity study:

1. Select $g$ reference parts ($g \ge 5$) whose master values $X_{m, 1} < X_{m, 2} < \dots < X_{m, g}$ span the full operating range of the gage.
2. Measure each part $m$ times ($m \ge 10$), computing the sample mean $\bar{Y}_i$ and observed bias $b_i = \bar{Y}_i - X_{m, i}$ for each reference part.
3. Fit a simple linear regression of bias on master reference value:

1. Test the null hypothesis $H_0: a_1 = 0$ (constant bias across operating range). If the slope $a_1$ is statistically significant ($p < 0.05$), the gage exhibits non-linearity.
2. The Percentage Linearity is defined as:

Acceptable industrial metrology standards require $\%\text{Linearity} \le 5\%$.

In complex mechanical assemblies—such as automotive engines, aircraft turbine stages, and robotic gearboxes—an overall critical clearance or assembly dimension $Y$ is a function of multiple individual component dimensions:

For linear stacks where dimensions add or subtract along a single kinematic chain:

where each component dimension $X_i$ is manufactured to an engineering specification $T_i = \pm \Delta_i$ centered at nominal dimension $m_i$. Mechanical designers must establish individual tolerances $\Delta_i$ to ensure that the final assembly clearance $Y$ conforms to its overall allowable specification $T_{\text{assembly}} = \pm \Delta_Y$.

Worst-Case Tolerance Synthesis (Arithmetic Stack-Up)

The traditional worst-case approach assumes that by extreme coincidence, all component parts simultaneously deviate to their maximum allowable extreme limits in the worst possible direction:

While worst-case tolerancing provides a $100\%$ mathematical guarantee that every possible combination of parts will assemble without interference, it represents an excessively pessimistic and economically punitive design philosophy. To satisfy a tight assembly tolerance $\Delta_Y$ under worst-case rules, individual part tolerances $\Delta_i$ must be set to minuscule values, forcing suppliers to employ slow, ultra-precision grinding and EDM machining, dramatically inflating unit manufacturing costs.

Statistical Tolerance Synthesis (Root-Sum-Square, RSS)

Statistical tolerance synthesis exploits the mathematical properties of the variance of linear combinations of independent random variables. If each component dimension $X_i$ is an independent random variable with mean $\mu_i = m_i$ and standard deviation $\sigma_i$:

Assuming each component manufacturing process operates at a common capability level $C_{pk}$ such that the natural tolerance spread corresponds to $\Delta_i = 3 C_{pk} \sigma_i$, we obtain the Root-Sum-Square (RSS) assembly tolerance:

Because $\sqrt{\sum a_i^2 \Delta_i^2} < \sum |a_i| \Delta_i$ for all $k \ge 2$, the statistical tolerance band is substantially narrower than the worst-case band. Conversely, for a fixed assembly requirement $\Delta_Y$, the statistical model allows designers to allocate significantly wider tolerances $\Delta_i$ to individual components:

This tolerance expansion dramatically lowers machining costs while maintaining assembly nonconformance rates at parts-per-million levels.

Dynamic Bender and Six-Sigma Stack Adjustments

In real manufacturing assemblies, component distributions may not be perfectly centered ($\mu_i \ne m_i$). Arthur Bender (1962) observed empirical assembly lines and proposed the Bender Factor of $1.5$:

Similarly, Motorola's Six Sigma design framework models potential long-term mean drift by allowing each component mean to drift by $\pm 1.5\sigma_i$. Under worst-case alignment of drifts, the assembly mean shifts by:

Combining this systematic shift with random RSS variance provides the foundation for robust Six Sigma mechanical tolerance synthesis.

When manufacturing data exhibit severe departures from Gaussian symmetry (such as non-negative dimensions with physical zero boundaries, surface roughness, or cycle times), transforming data via Norman L. Johnson's (1949) translation system provides a mathematically rigorous alternative to empirical percentiles.

The Johnson system maps an arbitrary continuous random variable $X$ to a standard normal variate $Z \sim \mathcal{N}(0, 1)$ through a monotonic transformation:

where $\gamma$ and $\eta > 0$ are shape parameters, $\xi$ is a location parameter, $\lambda > 0$ is a scale parameter, and $g(\cdot)$ is one of three functional forms:

1. Unbounded System ($S_U$): For distributions with infinite support in both directions:

2. Bounded System ($S_B$): For distributions with physical upper and lower boundaries (e.g., percentage measurements or dimensions bounded by tooling clearances $[\xi, \xi + \lambda]$):

3. Lognormal System ($S_L$): For strictly positive distributions bounded below by $\xi$:

Fitting and Capability Evaluation

Parameters are estimated by matching the first four empirical sample moments (mean $\bar{X}$, variance $S^2$, skewness $\sqrt{b_1}$, and kurtosis $b_2$) or using quantile matching across four selected percentiles ($z = \pm 0.5, \pm 1.5$). Once the optimal transformation function $f_J(X)$ is established, the engineering specification limits are mapped into the standard Gaussian space:

The non-normal process capability indices are evaluated directly in Gaussian units:

The nonconforming fraction is then computed with high numerical accuracy:

This methodology avoids the large sample variance inherent in nonparametric empirical percentile estimation, providing a robust industry benchmark for non-normal capability certification.

While $C_{pk}$ penalizes off-centering relative to specification boundaries and $C_{pm}$ penalizes off-target departures via quadratic loss, each index has distinct blind spots. A process can have a high $C_{pk}$ while being far from nominal target $T$ if specifications are asymmetric, and $C_{pm}$ does not indicate whether the mean is closer to the upper or lower specification limit.

To synthesize both philosophies into a single comprehensive metric, W. L. Pearn, Samuel Kotz, and Norman L. Johnson (1992) formulated the Unified Capability Index ($C_{pmk}$):

where $d = (USL - LSL)/2$ is the half-tolerance bandwidth and $M = (USL + LSL)/2$ is the specification midpoint.

Notice that the numerator of $C_{pmk}$ is identical to the numerator of $C_{pk}$ (measuring distance from the mean to the nearest specification limit), while its denominator is identical to the denominator of $C_{pm}$ (incorporating total mean-square error $\tau = \sqrt{\sigma^2 + (\mu - T)^2}$).

Relationships and Properties

1. If the process is perfectly centered on target ($\mu = M = T$):

1. If the process drifts away from target ($\mu \ne T$):

$C_{pmk}$ is strictly more sensitive to process off-centering than either $C_{pk}$ or $C_{pm}$, serving as the most stringent capability benchmark in high-precision aerospace and medical manufacturing.

The Six-Sigma Quality Benchmark ($Z_{\text{bench}}$)

In Motorola's Six Sigma methodology, process capability is expressed in terms of the Sigma Level ($Z_{\text{bench}}$). By definition, a Six Sigma process has:

Motorola recognized that over long-term production cycles, automated machinery inevitably experiences intermittent environmental drifts, tooling wear, and ambient temperature shifts. Empirical industrial studies established that process means typically drift by up to $\pm 1.5\sigma$ over time.

Under a worst-case $1.5\sigma$ sustained drift, the distance from the shifted mean to the nearest specification limit is:

The corresponding actual capability index drops to:

The defect rate under this $1.5\sigma$ shifted distribution is:

Thus, the world-class Six Sigma benchmark of $3.4\text{ PPM}$ explicitly incorporates a $1.5\sigma$ long-term mean drift into its statistical foundation!

Beyond empirical Gage R&R, international metrology calibration adheres to the ISO/IEC Guide to the Expression of Uncertainty in Measurement (GUM). Under GUM, measurement uncertainty is evaluated as:

1. Type A Evaluation: Method of evaluation of uncertainty by the statistical analysis of a series of observations (e.g., standard deviation of repeated readings $u_A = S / \sqrt{n}$).

2. Type B Evaluation: Method of evaluation of uncertainty by means other than statistical analysis (e.g., manufacturer calibration certificates, ambient thermal sensor drift, digital quantization error modeled as uniform distributions with standard uncertainty $u_B = a / \sqrt{3}$).

The Combined Standard Uncertainty $u_c(y)$ for an output measurement $Y = f(X_1, \dots, X_N)$ is:

The Expanded Uncertainty is $U = k \cdot u_c(y)$, where coverage factor $k = 2$ provides an approximate $95\%$ level of confidence, ensuring international traceability for critical industrial quality inspection.