Joint Distributions, Expectations and Conditional Variation

A single-variable distribution describes one numerical outcome. Many questions require several quantities together: whether two components pass, how a cost relates to a count, or how a response varies among groups. Marginal distributions alone do not answer these questions. Two models can give each variable the same individual distribution while assigning very different probabilities to their joint outcomes. A joint distribution records that missing relationship.

This unit develops joint, marginal and conditional distributions, independence of random variables, expected values of functions, covariance, and the decomposition of variation through conditional expectation. The reader should be able to move between a joint table and its margins, integrate a joint density over a specified region, compute an expectation with correct existence conditions, and explain why an overall variance contains both variation within groups and variation between group means.

The unit connects event calculations to numerical summaries. Independence remains a property of the full probability law, not of a scatterplot or a single covariance. Expectation is a probability-weighted average, not a value guaranteed to occur. Conditional expectation is a prediction based on a stated information set, not automatically a causal effect. Each formula will be paired with its assumptions so that the resulting summaries retain their meaning.

For discrete variables $X$ and $Y$, the joint PMF is $p_{X,Y}(x,y)=P(X=x,Y=y)$. Its entries are nonnegative and sum to one over the complete set of allowed pairs. A finite joint table places one variable's values along rows and the other's along columns. The table's orientation is a display choice, but its labels must remain consistent throughout a calculation.

The marginal PMF of $X$ is obtained by summing joint masses over every allowed value of $Y$. The marginal PMF of $Y$ is obtained by summing over $X$. These sums are justified because the contributing events are disjoint and exhaust the event for the retained variable. Marginalization does not assume independence. It is a direct consequence of additivity.

The word marginal originated in the placement of row and column totals, but the mathematical operation is not tied to a printed table. It discards one coordinate while preserving all its probability contributions. A missing cell cannot be treated as zero merely because it was omitted from a display. It must be impossible by the model or otherwise accounted for before the table is considered complete.

Different joint tables can have identical margins. For two binary variables with equal one-half margins, a table placing half the mass on each matching pair gives complete agreement of the variables. A table placing half on each mismatching pair gives complete opposition. A table placing one quarter on each of the four pairs gives independence. Every individual binary margin is the same in these three models, but their joint predictions differ radically.

This example explains why knowing two individual pass rates does not determine the probability that both components pass. Multiplying the rates adds an independence assumption. Adding them and subtracting an arbitrary overlap adds another assumption. A joint specification, a justified dependence model or defensible bounds is needed. Marginal percentages cannot reveal a missing relationship simply through more precise arithmetic.

When a variable has many possible values, a sparse joint representation can be convenient. Only pairs with nonzero masses need to be listed if the omitted pairs are explicitly assigned zero. The probability of a region is still the sum of all listed masses whose pairs satisfy its condition. A program should distinguish an explicitly zero pair from an unknown or unrecorded pair, especially when the representation comes from data rather than a fully specified model.

For an absolutely continuous pair, a joint density $f_{X,Y}$ assigns probability through area integrals. The density is nonnegative and integrates to one over the plane, with zero density outside the stated support. A probability for a region $R$ is the integral of the density over that region. As with a single-variable density, the value of the joint density at one point is not a point probability.

The support need not be a rectangle. A triangular or curved support creates integration limits depending on the other coordinate. Copying constant rectangular limits onto such a region can include impossible pairs or omit allowed ones. A sketch can help establish the limits, but the inequalities defining the region are the actual model specification. Numerical integration should use those same inequalities.

Marginal densities are obtained by integrating out the other variable. For a fixed value of $x$, integrate over all allowed $y$ at that value. If the support is triangular, the lower or upper limit can involve $x$. The resulting marginal density can vary even when the joint density is constant within its support. Constant density over a nonrectangular region does not imply uniform individual margins.

The joint CDF is $F_{X,Y}(x,y)=P(X\leq x,Y\leq y)$. It accumulates probability in a lower-left region. For a rectangle, probability can be expressed using four joint-CDF values with inclusion–exclusion. This is the two-coordinate counterpart of a single-variable interval formula. Endpoint atoms require appropriate one-sided limits in discrete or mixed cases, just as they did in Unit 3.

A joint distribution can combine discrete and continuous coordinates. For instance, a group label is discrete while a measurement within a group may be continuous. One convenient representation specifies group probabilities and conditional measurement densities. The resulting marginal measurement density is their weighted sum. A single ordinary two-dimensional density with respect to area is not the appropriate representation when one coordinate is categorical.

For a discrete value $y$ with $p_Y(y)>0$, the conditional PMF of $X$ given $Y=y$ is the corresponding joint column divided by its total:

The conditional masses sum to one within that column. Row conditioning follows the same principle with the labels exchanged. A table may contain a row or column of total zero. The elementary conditional distribution there is undefined, since the model provides no positive-probability reference collection to normalize. Returning an arbitrary uniform distribution in that case would invent information.

For a pair with a joint density, a conditional density can be specified for relevant values with positive marginal density by

This formula is a conditional-density construction. It is not obtained by dividing the zero point probability of a pair by the zero point probability of one continuous coordinate. Its justification comes from the joint-density factorization and integration identities. Conditional densities are defined up to appropriate marginal-null sets; isolated values without marginal weight need not have a uniquely determined conditional law.

A useful elementary interpretation considers a thin band of $Y$ values near a given value. Under regularity conditions, the distribution of $X$ within such bands approaches the conditional density. This provides intuition but is not a substitute for checking the density formula and its support. In particular, a denominator evaluated outside the marginal support cannot create a meaningful conditional law through a formal ratio.

The support of a conditional distribution can change with the conditioning value. If $0<X<Y<1$, then given an interior value of $Y$, the allowed $X$ values lie between zero and that value. A conditional density uniform on that changing interval does not imply that the unconditional $X$ distribution is uniform on the whole unit interval. Averaging across conditioning values reweights the overlapping intervals and produces a different margin.

Conditional distributions recover the joint distribution when combined with the conditioning margin. In the discrete case, multiply conditional masses by the margin. In the continuous case, multiply conditional densities by the marginal density where the factorization is defined. Averaging a conditional event probability over the conditioning law recovers the event's unconditional probability. This is the distributional form of the law of total probability.

Two random variables are independent when their events factorize across all relevant sets. In terms of their CDFs, independence means $F_{X,Y}(x,y)=F_X(x)F_Y(y)$ for every pair of real thresholds. For discrete variables, factorization of the joint PMF into the two marginal PMFs at all pairs is equivalent. For a pair with densities, factorization of the joint density into marginal densities almost everywhere provides the corresponding criterion.

Checking one joint cell does not establish independence. Checking one tail event does not establish it either. The criterion concerns the whole law. For a small finite table, every cell can be compared with the product of its row and column margins. For a parametric density, a symbolic factorization plus a correct support analysis can establish the property. A nonrectangular support often signals dependence when the marginal supports would permit pairs forbidden by the joint law.

If variables are independent, measurable functions of them remain independent. Events concerning the transformed variables correspond to events concerning the original variables, and the original factorization applies. This permits independent components to be rescaled, thresholded or otherwise transformed without creating cross-component dependence. It does not imply that two functions of the same random variable are independent. Sharing a common input can generate strong dependence.

Independence also justifies products of expectations under appropriate integrability conditions. If independent variables have finite absolute first moments, their product is integrable and its expectation factorizes. Without independence, factorization generally fails. Without the existence conditions, an expression involving undefined or infinite signed expectations may be meaningless. A formula should not be used simply because its algebra resembles multiplication of ordinary numbers.

For a discrete variable with finite absolute first moment, expectation is

For an absolutely continuous variable under the corresponding integrability condition, expectation is $\int x f_X(x)\,dx$. A nonnegative variable can have an extended expectation equal to positive infinity. A signed variable requires care: if both its positive and negative contributions are infinite, their difference is undefined. Pairing terms to force cancellation does not create a legitimate finite expectation.

The mean need not be a possible value. A binary variable with probability one third of one has expectation one third, even though it takes only zero or one. The mean summarizes probability-weighted location; it does not predict that each realization will equal that summary. Whether it is useful as a forecast depends on the loss criterion and the application. Under squared error it has an optimality property developed below.

Expectation of a function can be computed directly from the distribution of the original variable:

in the discrete case, with an integral for the continuous case. One need not first derive the entire distribution of the transformed variable. This is often called the law of the unconscious statistician, but the descriptive name is less important than the mechanism: each original value contributes its probability weight multiplied by the transformed numerical value. The same existence conditions apply to the transformed quantity.

For a function of two variables, use the joint law. A discrete expectation sums $g(x,y)p_{X,Y}(x,y)$ across all pairs; a continuous expectation integrates the function against the joint density. Marginal distributions alone suffice only for functions that separate in a way justified by linearity or independence. The expectation of a product generally requires joint information.

An indicator variable $I_A$ equals one on event $A$ and zero elsewhere. Its expectation is $P(A)$. This identity converts event calculations into numerical expectations and makes counting arguments efficient. A number of occurrences can be represented as a sum of indicators. Its expectation is the sum of the individual event probabilities even when the indicators are dependent. Dependence matters for variance and higher-order properties, but not for this linearity identity.

Expectation is linear for integrable variables. For constants $a$ and $b$, $E[aX+bY]=aE[X]+bE[Y]$. Independence is not required. The proof expands the weighted sum or integral and separates its terms under the relevant integrability conditions. For a finite collection, the result extends by repeated application. Infinite sums require additional justification for exchanging summation and expectation; a finite identity should not be applied blindly to an infinite series.

Variance is the expected squared deviation from the mean, assuming a finite second moment:

The second identity follows by expanding the square and using linearity. The first form makes nonnegativity explicit. The second can be convenient for exact calculations but may suffer subtractive cancellation in floating-point computation when the two large terms are nearly equal. Stable numerical algorithms can accumulate centered differences or use carefully derived online updates.

The standard deviation is the nonnegative square root of variance. It has the variable's original units, while variance has squared units. Adding a constant to a variable does not change its variance. Multiplying by a constant multiplies variance by the square of that constant. These transformation rules help check units and scaling. A change from meters to centimeters multiplies numerical standard deviation by one hundred and numerical variance by ten thousand.

Finite second moment implies finite absolute first moment, so the mean used in a finite variance calculation is well defined. The converse is false: a distribution can have a finite mean and infinite variance. A formula reporting a finite variance must therefore be justified by the distribution's second moment, not merely by the existence of a central average. Later transform and tail discussions will emphasize these existence differences.

For variables with finite second moments, covariance is

Positive covariance indicates that their centered deviations tend to have matching signs in this weighted average. Negative covariance indicates opposite signs on average. Zero covariance means that this particular average is zero; it does not mean the joint distribution factorizes. Dependence can be nonlinear or arranged so that positive and negative products cancel.

Expanding the square of a centered sum gives

Subtracting a variable changes the sign of the covariance term. More generally, scaling each variable scales their covariance by the product of the constants. For a collection of variables, variance of a sum includes every individual variance and every distinct pair covariance. Omitting covariance terms adds an uncorrelatedness assumption. Independence is a sufficient condition for their vanishing when the moments exist, but it is stronger than necessary.

Correlation normalizes covariance by the two positive standard deviations. It is dimensionless and lies between minus one and one. The bound follows from the nonnegativity of expected squared linear combinations, or from the Cauchy–Schwarz inequality applied to centered variables. If either variance is zero, the usual correlation coefficient is undefined because its denominator vanishes. Calling it zero would confuse an undefined normalization with an uncorrelated pair.

A simple nonlinear example separates dependence from covariance. Let $X$ be uniform on the interval from minus one to one and set $Y=X^2$. Symmetry gives zero mean for $X$ and zero expectation for $X^3$, so covariance of $X$ and $Y$ is zero. Nevertheless $Y$ is completely determined by $X$, and the variables are dependent. Observing a large value of $Y$ constrains the magnitude of $X$. A zero covariance cannot reveal that relationship by itself.

Covariance also depends on measurement choices. Multiplying one variable by a positive factor changes the covariance magnitude while preserving its sign; multiplying by a negative factor reverses the sign. Correlation removes positive scale factors but remains a summary of linear association. Neither summary alone establishes a causal connection or guarantees that a straight-line model predicts well across the whole range.

For a discrete conditioning value with positive mass, conditional expectation is the mean under the conditional PMF. For a continuous conditioning variable with a suitable conditional density, it is the integral under that density. Write $m(y)=E[X\mid Y=y]$. Substituting the random variable $Y$ into this function gives the random variable $m(Y)$, commonly written $E[X\mid Y]$. It varies with the information supplied by $Y$.

Conditional expectation is not the same as plugging a conditioning value into an unconditional formula. The conditional distribution must first be specified or derived. If the allowed range of $X$ changes with $Y$, that range affects the conditional mean and variance. A density formula copied onto the unconditional support can give an apparently neat answer that does not describe the conditional experiment.

The law of iterated expectation states

for integrable $X$. In the discrete case, substitute the conditional mean, multiply by the marginal weight of each conditioning value, and use the joint factorization. The two sums combine into the full joint expectation of $X$. In the continuous case, the same argument uses integrals and the conditions permitting their interchange. The result is averaging within information groups and then averaging across them.

The outer expectation uses the actual distribution of the conditioning variable. It is not generally a simple arithmetic average of group means. If one group has probability one quarter and another three quarters, their means receive those weights. This distinction is the probability counterpart of combining unequal-size group averages in descriptive statistics. Changing the experiment's group weights changes the overall expectation even if every within-group mean stays fixed.

Conditional expectation of a centered residual is zero: $E[X-m(Y)\mid Y]=0$. This identity says that after using the conditional mean, no average residual remains within the information groups under the model. It does not imply every residual is zero. It also does not establish that the real data follow the model. A conditional mean can be an exact model property while the observed process violates its assumptions.

For a variable with finite second moment, total variance is

The first term is average within-information variance. The second is variation of the conditional means across the information distribution. Both are nonnegative. An overall variance cannot therefore be obtained by averaging only within-group variances unless the conditional mean is constant almost surely. Differences among group means are an additional source of variation.

To derive the identity, let $\mu=E[X]$ and $m(Y)=E[X\mid Y]$. Decompose the centered variable as $X-\mu=(X-m(Y))+(m(Y)-\mu)$. Squaring produces two squared terms and a cross term. Conditional on $Y$, the second factor in the cross term is fixed while the first has conditional mean zero. Its overall expectation is therefore zero. The first squared term averages to conditional variance, and the second is the variance of the conditional mean.

The decomposition is sometimes described as within-group and between-group variance. That wording is useful for a categorical conditioning variable but should not suggest that the groups must be causal categories. Conditioning on any suitable information variable can produce the identity. The between component measures variation explained by that information in a squared-error sense, not a causal effect of manipulating it.

The law also explains why heterogeneity can create marginal dependence. Two outputs conditionally independent given a shared group can have a covariance caused by their varying group means. A corresponding total-covariance identity is $\operatorname{Cov}(X,Z)=E[\operatorname{Cov}(X,Z\mid Y)]+\operatorname{Cov}(E[X\mid Y],E[Z\mid Y])$ under the needed finite second moments. Conditional independence eliminates the first term, while the second may remain positive, negative or zero.

Expectation has a prediction interpretation under squared loss. Among constant predictions $a$ for a variable with finite second moment, the mean minimizes $E[(X-a)^2]$. Expanding around the mean gives variance plus $(E[X]-a)^2$. The second term is minimized by choosing the mean. This establishes an optimality property for that particular loss; it does not make the mean optimal for every practical objective.

If the prediction may depend on an observed variable $Y$, its conditional mean minimizes mean squared error among square-integrable functions of that information. Let a candidate predictor be $h(Y)$. Decompose the error as $X-h(Y)=(X-m(Y))+(m(Y)-h(Y))$. The expected cross term is zero by the conditional-residual identity. The resulting error is average conditional variance plus $E[(m(Y)-h(Y))^2]$. The second term cannot be negative and vanishes when the predictor agrees with the conditional mean almost surely.

This property is about available information and loss. A predictor using only $Y$ cannot remove the average conditional variation left after $Y$ is known. More information can reduce the optimal expected squared error, but merely adding irrelevant or incorrectly modeled information does not guarantee improvement in an estimated predictor. The exact probability result describes the true conditional law within the model, while fitting that law from finite data introduces statistical uncertainty.

An optimal squared-error predictor can produce values outside the variable's discrete support, just as an unconditional expectation can. Predicting a binary result by its conditional probability is meaningful as a squared-error forecast even though the observed result is zero or one. If a hard classification is required, a decision rule and its error costs must be specified separately. A forecast probability is not automatically a selected class.

An expectation of a sum is easy to obtain through linearity, but the complete distribution of a sum requires more information. For discrete variables, the event that a sum equals a particular value is a union of joint outcomes whose coordinates add to that value. Therefore its mass is obtained by adding the joint masses along those pairs. This calculation uses the joint law and does not assume independence.

If the variables are independent, each joint mass is a product of its marginal masses. The sum distribution then becomes a convolution: $p_{X+Y}(s)=\sum_x p_X(x)p_Y(s-x)$. The sum ranges over supported values, and unsupported arguments of either PMF contribute zero. This representation is useful because it derives a new distribution from two known independent laws without constructing a large joint table explicitly.

Support restrictions remain important. If one variable can take only zero or two and another only zero or three, their sum can take zero, two, three and five. Intermediate integers are not automatically possible. A numerical convolution routine operating on evenly spaced arrays must represent those missing values by zero rather than compressing the arrays and accidentally treating adjacent indices as adjacent numerical values. Computational storage conventions can otherwise change the modeled quantity.

In the continuous independent case, a sum density can be expressed as $f_{X+Y}(s)=\int f_X(x)f_Y(s-x)\,dx$ under the appropriate density assumptions. The effective integration range is the set of values where both density arguments lie in their supports. A pair of uniform variables on bounded intervals produces a changing overlap length and generally does not produce another uniform sum. The geometry of that overlap explains the familiar rise and fall of the resulting density.

The convolution formula is not valid for arbitrary dependent variables if the joint density is replaced by the product of margins. For a pair with a joint density, the corresponding integral uses $f_{X,Y}(x,s-x)$ after the appropriate transformation. The CDF can also be obtained by integrating the joint law over the region where the coordinate sum is below the threshold. That event-based method makes the need for a joint specification explicit and can be easier for unusual supports.

There is a clear contrast between distribution and expectation here. Linear expectation needs only the individual means when the relevant integrability conditions hold. The distribution of the sum needs the dependence structure, and its variance needs covariance. Three levels of description therefore require different information. A correct answer about one level cannot automatically answer the others. Knowing a sum's mean is not enough to estimate a threshold probability without additional distributional information or a justified bound.

Conditional expectation obeys linearity within a fixed information set. A function of the conditioning variable is known under that information, so it can be taken outside a conditional average when the expressions are integrable. In particular, $E[h(Y)X\mid Y]=h(Y)E[X\mid Y]$. This identity supports the zero-cross-term arguments used for total variance and squared-error prediction. Its interpretation is straightforward: a quantity already determined by the information is a constant during averaging over the remaining uncertainty.

Nonlinear functions do not generally commute with expectation. The conditional expectation of a square is not the square of the conditional expectation. Their difference is the conditional variance: $E[X^2\mid Y]-(E[X\mid Y])^2=\operatorname{Var}(X\mid Y)$. This quantity is nonnegative, showing directly why squaring an average usually understates the average squared value. Equality requires no residual variation under the given information, up to sets of probability zero.

The same distinction applies to unconditional averages. A population with equal probabilities on zero and two has mean one and second moment two. Squaring the mean gives one rather than two. The missing difference is variance. This simple calculation is a useful diagnostic when a nonlinear prediction is produced by applying a function to an average input. Such a prediction may differ from the average outcome under the full input distribution.

The total-covariance law can be derived without memorizing a separate formula. Write each centered variable as its residual after conditioning plus its conditional mean minus the overall mean. Multiply the two decompositions. Averaging the residual product within the information set gives conditional covariance. Each mixed residual-and-conditional-mean term has expectation zero because the residual has conditional mean zero. The final mean-product term is the covariance of the conditional means.

This derivation also reveals why conditional independence is sufficient but not necessary for the residual covariance term to vanish. Conditional uncorrelatedness can eliminate that term even when the residuals have a nonlinear dependence. Conversely, conditional independence cannot eliminate covariance among the conditional means. Distinguishing the two terms prevents a common mistake in mixture models: declaring outputs independent after checking only their within-group mechanisms.

Conditional covariance can have different signs across groups. Their weighted average might cancel, while the between-mean covariance remains. A zero overall covariance can therefore arise through several mechanisms and does not identify one unique dependence structure. Examining only the final scalar summary discards information contained in conditional distributions. A joint-law or stratified analysis may be needed to understand the relationship relevant to a practical decision.

The variance of the conditional mean cannot exceed total variance because the remaining term is nonnegative. It equals total variance only when the conditional variance averages to zero. For a nonnegative conditional variance, a zero average implies it is zero almost surely. In that limiting case the information determines the variable up to model-null sets. At the opposite extreme, a constant conditional mean yields zero between-mean variance, though other kinds of dependence on the information can still remain through conditional spread or shape.

Expectation respects ordering when the relevant expectations are defined: if one variable is at most another almost surely, its expectation is no greater. For a nonnegative variable and a positive threshold, the event that the variable reaches the threshold implies that its value contributes at least that threshold on the event. This yields Markov's inequality, $P(X\geq a)\leq E[X]/a$. Applying it to the squared centered variable gives Chebyshev's inequality, $P(|X-E[X]|\geq t)\leq\operatorname{Var}(X)/t^2$ for a positive threshold and finite variance.

These bounds require no normality assumption, but they can be loose. A bound larger than one is improved by the trivial probability upper bound one. A bound is not an exact probability and should not be plotted or reported as though it were a distribution prediction. Its value is that limited moment information can still constrain event probabilities. Later model-specific laws often give more informative exact probabilities because they supply more information than a mean and variance.

Conditional expectation offers an optimal prediction only relative to the stipulated law and squared loss. It does not guarantee a small error, and it does not remove uncertainty about an estimated model. A mixture with widely separated means may offer a much better prediction after the group is observed, while substantial within-group variation remains. Reporting both predicted location and remaining conditional variance helps describe that residual uncertainty.

Independent computational checks should use a route different from the written derivation where feasible. A finite joint table can be enumerated into a probability-weighted list of pairs, then its means, variances and conditional slices can be computed directly. A continuous triangular model can be integrated in both coordinate orders and checked with numerical quadrature. A mixture can be checked by its expanded marginal law as well as the conditional decomposition. Agreement verifies the encoded mathematical assumptions; it does not validate their real-world applicability.

A joint-table explorer should show row totals, column totals and the selected conditional distribution together. Editing a cell must either preserve normalization through an explicit rule or require the user to normalize. Negative cells and a zero total must be rejected or explained. A conditional slice with zero marginal weight should display an undefined-condition message rather than fabricated percentages. Accessible labels should identify whether an output is a joint mass, a marginal probability or a conditional probability.

The main lesson is that numerical summaries are derived from full probability laws and information sets. Linearity does not require independence. Product expectations generally do. Zero covariance is weaker than independence. Conditional means vary with the available information, and overall variance combines residual variation with variation of those means. The next unit studies moments and generating functions as compact tools for representing these properties, while retaining the same existence conditions and model discipline.