Independence and product measures

The lesson "Probability spaces and Kolmogorov's axioms" turned a probability into a measure of total mass 1 and showed that the binary digits of one uniform number behave like coin tosses, and "Expectation, variance and inequalities" turned averages into integrals and measured spread by the variance. Both looked at one random quantity at a time, but real questions involve several: two dice, the horizontal and vertical position of a dart, ten coins. This lesson makes the idea that one quantity carries no information about another precise under the name independence, first for events, then for σ-algebras and random variables. It shows that independence is the same thing as area being length times length, which leads to product measures and to Fubini's theorem for integrals over a product. From these come the rule $\mathbb{E}[XY] = \mathbb{E}[X]\,\mathbb{E}[Y]$, the covariance, the variance of a sum, and finally the proof, promised earlier, that the binary digits of a uniform number are independent fair coins.

Conditional probability in brief

Roll two fair dice and suppose you are told that the first shows 3. The 36 equally likely outcomes shrink to the six with first die 3, and the sum is 8 exactly when the second die shows 5. So the probability of a sum of 8 has become $1/6 \approx 0.1667$, while before the information it was $5/36 \approx 0.1389$. Conditioning is this shrinking: keep only the outcomes of the event $B$ known to have happened, and rescale so that $B$ carries total probability 1. For an event $B$ with $\mathbb{P}(B) > 0$, the conditional probability of $A$ given $B$ is

In words: of the probability that $B$ carries, the fraction that lies inside $A$. For the dice, $A$ is "the sum is 8" and $B$ is "the first die shows 3". The intersection is the single outcome $(3, 5)$, so $\mathbb{P}(A \cap B) = 1/36$, and $\mathbb{P}(B) = 1/6$; the quotient is $1/6$. The division by $\mathbb{P}(B)$ is the rescaling; without it the restricted probabilities would add up to $\mathbb{P}(B)$. For a fixed $B$ the map $A \mapsto \mathbb{P}(A \mid B)$ obeys the axioms of "Probability spaces and Kolmogorov's axioms": it is a probability measure again, the old one restricted to $B$ and rescaled. This lesson needs conditional probability only as far as independence requires it; a later lesson of this track treats conditioning in full.

Independent events

Two events are independent when learning that one of them happened does not change the probability of the other: $\mathbb{P}(A \mid B) = \mathbb{P}(A)$. Substitute the definition of conditional probability and multiply by $\mathbb{P}(B)$, and the condition becomes the product rule

The product rule is taken as the definition, because it needs no assumption $\mathbb{P}(B) > 0$ and treats the two events symmetrically. It says that the share of the whole space taken by "both" equals the share of $A$ times the share of $B$, exactly as the area of a rectangle is width times height.

Independence does not require two separate physical mechanisms. Take one fair die, $A =$ "even" and $B = \{1, 2\}$. Then $\mathbb{P}(A) = 1/2$, $\mathbb{P}(B) = 1/3$ and $A \cap B = \{2\}$ has probability $1/6 = \tfrac12 \cdot \tfrac13$. The events are independent, because the outcomes in $B$ are half even, the same fraction as in the whole die. Replace $B$ by $\{1, 2, 3\}$ and the property is gone: $\mathbb{P}(A \cap B) = 1/6$, but $\mathbb{P}(A)\mathbb{P}(B) = 1/4$. Independence is a numerical relation between proportions inside the probability space. It usually arises because the events concern different dice or coins, but the definition only looks at the numbers.

An event of probability 0 or 1 is independent of every event, since both sides of the product rule are then 0 or $\mathbb{P}(A)$. Independence also survives taking complements. If $A$ and $B$ are independent, so are $A$ and $B^c$:

The first step is the rule $\mu(A \setminus B) = \mu(A) - \mu(A \cap B)$ from "Measuring the size of sets", the second is the product rule for $A$ and $B$, and the third is $\mathbb{P}(B^c) = 1 - \mathbb{P}(B)$. Applying the same step to $A^c$ and $B$, and then to $A^c$ and $B^c$, shows that all four pairs made of $A$, $A^c$ and $B$, $B^c$ satisfy the product rule.

Disjoint is not the same as independent; it is nearly the opposite. If $A$ and $B$ are disjoint and both have positive probability, then $\mathbb{P}(A \cap B) = 0$ while $\mathbb{P}(A)\mathbb{P}(B) > 0$. Learning that $B$ happened rules $A$ out completely, which is the strongest possible dependence.

Pairwise and mutual independence

With three or more events, independence of every pair is not enough. Toss two fair coins, so that the four outcomes HH, HT, TH, TT have probability $1/4$ each, and let

• $A$ = the first coin is heads $= \{HH, HT\}$,
• $B$ = the second coin is heads $= \{HH, TH\}$,
• $C$ = both coins show the same side $= \{HH, TT\}$.

Each of the three events has probability $1/2$. Each pairwise intersection is the single outcome $HH$, with probability $1/4 = \tfrac12 \cdot \tfrac12$, so $A, B$, then $A, C$ and $B, C$ are all independent pairs. But the triple intersection $A \cap B \cap C = \{HH\}$ also has probability $1/4$, while the product of the three probabilities is $1/8$. The reason: if $A$ and $B$ both happened, both coins are heads and $C$ is certain, although neither $A$ nor $B$ alone says anything about $C$.

Events $A_1, \ldots, A_n$ are therefore called mutually independent if the product rule holds for every sub-collection:

For three events this means four equations, three for the pairs and one for the triple; for $n$ events it means $2^n - n - 1$. An infinite family is mutually independent when every finite sub-collection is. When a sequence of coin tosses is called independent, this is what is meant, and from here on "independent" without further qualification means mutually independent. Events that are independent only in pairs are called pairwise independent.

Independent σ-algebras and random variables

Often one wants to compare everything that can be asked about one quantity with everything that can be asked about another: the first die against the second, the horizontal position of a dart against its vertical position. Recall from "σ-algebras and Borel sets" that a σ-algebra is a collection of questions that can be answered, closed under complements and countable unions. Two σ-algebras $\mathcal{G}$ and $\mathcal{H}$ inside the event collection $\mathcal{F}$ are independent if every question of one is independent of every question of the other:

Independence of two events $A$ and $B$ is the special case of the smallest σ-algebras $\{\emptyset, A, A^c, \Omega\}$ and $\{\emptyset, B, B^c, \Omega\}$: the pairs involving $A$, $A^c$, $B$ and $B^c$ were settled in the previous section, and $\emptyset$ and $\Omega$ have probabilities 0 and 1.

A random variable $X$ comes with the σ-algebra $\sigma(X)$ of "Measurable functions", made of all events $\{X \in B\}$ for Borel sets $B$, the questions that can be answered by looking at $X$ alone. Random variables $X$ and $Y$ are independent if $\sigma(X)$ and $\sigma(Y)$ are independent, that is, if

Checking every pair of Borel sets looks impossible, but half-lines are enough. The variables $X$ and $Y$ are independent exactly when

where $F_X$ and $F_Y$ are the distribution functions of "Random variables and their distributions". The reason is the argument that made length unique in "Lebesgue measure and null sets": half-lines are closed under intersection and generate the Borel sets, and the sets for which a product rule holds are closed under complements and countable disjoint unions, so a product rule that holds on half-lines holds on all Borel sets. This is an idea of the proof, applied once to each variable, not a full proof. For the two dice, $\mathbb{P}(X \le 3, Y \le 2) = 6/36 = 1/6$, and $F_X(3)F_Y(2) = \tfrac12 \cdot \tfrac13 = 1/6$. For variables that take countably many values, the condition reads $\mathbb{P}(X = i, Y = j) = \mathbb{P}(X = i)\,\mathbb{P}(Y = j)$ for all values $i, j$.

Independence passes through functions. If $X$ and $Y$ are independent and $g$, $h$ are measurable functions, then $g(X)$ and $h(Y)$ are independent. The reason is that an event about $g(X)$ is an event about $X$: $\{g(X) \in B\} = \{X \in g^{-1}(B)\}$, and the preimage $g^{-1}(B)$ is a Borel set because $g$ is measurable. So $\sigma(g(X)) \subseteq \sigma(X)$, and a smaller σ-algebra of an independent pair stays independent. The parity of the first die and the square of the second die are independent for this reason. Finally, random variables $X_1, \ldots, X_n$ are mutually independent if $\mathbb{P}(X_1 \in B_1, \ldots, X_n \in B_n) = \prod_i \mathbb{P}(X_i \in B_i)$ for all Borel sets, which is the same as mutual independence of the σ-algebras $\sigma(X_i)$.

Product measures

The area of a rectangle $[0, a] \times [0, b]$ in the unit square is $ab$: length times length. "Lebesgue measure and null sets" built area by covering sets with rectangles. The same recipe works for any two measure spaces. Take $(\Omega_1, \mathcal{F}_1, \mu_1)$ and $(\Omega_2, \mathcal{F}_2, \mu_2)$ and form the set $\Omega_1 \times \Omega_2$ of pairs $(\omega_1, \omega_2)$. A rectangle is a set $A \times B$ with $A \in \mathcal{F}_1$ and $B \in \mathcal{F}_2$. The product σ-algebra $\mathcal{F}_1 \otimes \mathcal{F}_2$ is the σ-algebra generated by the rectangles, in the sense of "σ-algebras and Borel sets"; for $\mathbb{R} \times \mathbb{R}$ with the Borel sets it is the Borel σ-algebra of the plane. The product measure $\mu_1 \otimes \mu_2$ is the measure on it that gives every rectangle its natural size,

Such a measure exists, by covering a set with rectangles and taking the smallest total, and it is unique for finite measures and for measures such as length on $\mathbb{R}$, which are countable sums of finite pieces. The uniqueness argument is the one used for length: the intersection of two rectangles, $(A \times B) \cap (A' \times B') = (A \cap A') \times (B \cap B')$, is again a rectangle, so a measure is pinned down by its values on rectangles. Two examples. Counting measure on the six faces of a die, taken twice, gives counting measure on the 36 pairs, and with the uniform probability $1/6$ on each face every pair has probability $\tfrac16 \cdot \tfrac16 = 1/36$. Length taken twice on $[0, 1]$ gives area on the unit square.

Now the link to probability. Two random variables $X$ and $Y$ on the same probability space have a joint distribution, the measure on the plane $\mathbb{P}_{(X, Y)}(D) = \mathbb{P}((X, Y) \in D)$, which is the image measure of $\omega \mapsto (X(\omega), Y(\omega))$ in the sense of "Measurable functions". On a rectangle it reads $\mathbb{P}_{(X, Y)}(B \times C) = \mathbb{P}(X \in B, Y \in C)$, and the definition of independence says that this equals $\mathbb{P}_X(B)\,\mathbb{P}_Y(C)$. Since rectangles determine a measure, the two statements are the same:

In words, the joint distribution of independent variables is the product of their separate distributions, and nothing else can be said about a pair unless the pair is told to be independent. If $X$ and $Y$ have densities $f_X$ and $f_Y$, the joint distribution of an independent pair has density $f_X(x)\,f_Y(y)$, because $\mathbb{P}(X \le a, Y \le b) = F_X(a)F_Y(b)$ is the integral of $f_X(x)f_Y(y)$ over the quadrant. Two independent uniform numbers on $[0, 1]$ form exactly a point chosen by area on the unit square. The dart of "Probability spaces and Kolmogorov's axioms", thrown uniformly at the square, therefore has independent horizontal and vertical coordinates.

The joint distribution holds more information than the two separate distributions. The figure shows a point $(X, Y)$ in the unit square in which both coordinates are uniform on $[0, 1]$, whatever the dependence. The event $A = \{X \le a\}$ is the vertical strip of width $a$, the event $B = \{Y \le b\}$ is the horizontal strip of height $b$, and $A \cap B$ is the rectangle $[0, a] \times [0, b]$ in the lower-left corner. The shading is the joint density $c(x, y) = 1 + \theta(1 - 2x)(1 - 2y)$, so that $\theta$ sets how strongly the coordinates are tied together. Start at $\theta = 0$: the shading is flat, the pair is two independent uniforms, and the rectangle with $a = 0.6$ and $b = 0.5$ holds $\mathbb{P}(A \cap B) = 0.3 = 0.6 \cdot 0.5$. Move $\theta$ to $1$: probability moves toward the lower-left and upper-right corners, the same rectangle now holds $0.36$, and $\mathbb{P}(A \mid B)$ rises from $0.6$ to $0.72$, so learning that $Y$ is small makes $X$ small more likely. At $\theta = -1$ the rectangle holds $0.24$ and $\mathbb{P}(A \mid B) = 0.48$. Throughout, $\mathbb{P}(A)$ and $\mathbb{P}(B)$ stay at $0.6$ and $0.5$: the dependence is entirely in how the coordinates are tied together. (The family is known as Farlie–Gumbel–Morgenstern; the name is not needed.)

Fubini and Tonelli

A product measure lets you integrate over a product, and the way to do it is to slice. To find the area of a region in the square, add up the lengths of its vertical slices. In general, to integrate a function $f(x, y)$ over the product, hold $x$ fixed, integrate over $y$, and integrate the result over $x$. For $f(x, y) = xy$ on the unit square,

Doing the $x$-integral first gives $\int_0^1 y \cdot \tfrac12\,dy = \tfrac14$ as well. Two theorems say when the order does not matter. Tonelli's theorem: for a measurable $f \ge 0$ on the product, both iterated integrals exist, possibly infinite, and both equal $\int f \,d(\mu \otimes \nu)$. Fubini's theorem: the same holds for every integrable $f$, that is, when $\int |f|\,d(\mu \otimes \nu) < \infty$. The proof builds up in the order of "The Lebesgue integral": for the indicator of a rectangle the statement is the product rule; the uniqueness argument extends it to every set of the product σ-algebra; linearity gives simple functions; monotone convergence, from "Convergence theorems", gives non-negative functions; and $f = f^+ - f^-$ gives integrable ones. This is an outline, not a proof.

Tonelli's theorem also explains a formula from earlier. Apply it to the indicator $\mathbf{1}\{t < f(x)\}$ of the region under the graph, on the product of the space and $[0, \infty)$. Integrating over $t$ first gives $f(x)$, integrating over $x$ first gives $\mu(f > t)$, so $\int f\,d\mu = \int_0^\infty \mu(f > t)\,dt$, the layer-cake formula of "The Lebesgue integral". With $\mathbb{P}$ in place of $\mu$ it is the tail formula of "Expectation, variance and inequalities".

Now the consequence for probability. Let $X$ and $Y$ be independent and integrable. The expectation of $XY$ is the integral of the function $(x, y) \mapsto xy$ against the joint distribution, which is the product measure $\mathbb{P}_X \otimes \mathbb{P}_Y$. Tonelli applied to $|xy|$ gives $\mathbb{E}|XY| = \mathbb{E}|X|\,\mathbb{E}|Y| < \infty$, so $XY$ is integrable and Fubini applies:

For the two dice $\mathbb{E}[XY] = 3.5 \cdot 3.5 = 12.25$, which a sum over the 36 pairs confirms. Without independence the rule fails: take $Y = X$, and $\mathbb{E}[X^2]$ is larger than $\mathbb{E}[X]^2$ by the variance. The gap is exactly what the next section measures.

Integrability cannot be dropped from Fubini's theorem. The function $f(x, y) = (x^2 - y^2)/(x^2 + y^2)^2$ on the unit square has the two iterated integrals $\pi/4$ and $-\pi/4$ in the two orders; it is not integrable, since $\int |f|$ is infinite near the origin.

Covariance and correlation

For independent variables $\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y]$. The size of the gap measures how far a pair is from behaving like that. For $X$ and $Y$ with finite variance, the covariance is

The two forms agree after expanding the product and using linearity of expectation. The second form shows what covariance records: it is positive when above-average values of $X$ tend to come with above-average values of $Y$, and negative when they come with below-average ones. Taking $Y = X$ gives $\operatorname{Cov}(X, X) = \operatorname{Var} X$. Independence gives $\operatorname{Cov}(X, Y) = 0$ by the product rule for expectations.

Covariance is exactly what is missing in the variance of a sum. Write $\tilde X = X - \mathbb{E}X$ and $\tilde Y = Y - \mathbb{E}Y$; the centred sum is $\tilde X + \tilde Y$. Then

The first line squares the sum and uses linearity; the second recognises the three expectations. For independent variables, or just uncorrelated ones, the covariance term vanishes and the variances add. The same holds for any number of pairwise uncorrelated variables.

Ten dice show how much this matters. One die has variance $35/12 \approx 2.917$ and standard deviation $1.708$. The sum of ten independent dice has variance $10 \cdot 35/12 \approx 29.17$ and standard deviation $\sqrt{29.17} \approx 5.401$. If the ten dice were instead the same die copied ten times, the sum would be $10X$ with standard deviation $10 \cdot 1.708 = 17.08$. Independent fluctuations partly cancel: a high value of one die is likely to meet a low value of another, so the spread of the sum grows like $\sqrt{n}$ instead of $n$. The average of the ten dice has standard deviation $1.708/\sqrt{10} \approx 0.5401$, a little under a third of the spread of a single die.

The result of the third and fourth steps looks surprising next to the first section, where learning the first die changed the probability of a sum of 8. A sum of 7 is special: whatever the first die shows, exactly one value of the second makes the sum 7, so the first die tells you nothing about whether the sum is 7. The sum of 8 is impossible after a first die of 1, and that is a piece of information.

Covariance depends on the units of measurement. The correlation removes them by dividing by the two standard deviations:

Correlation always lies in $[-1, 1]$. This is the Cauchy–Schwarz inequality, $\mathbb{E}|\tilde X \tilde Y| \le \sqrt{\mathbb{E}[\tilde X^2]\,\mathbb{E}[\tilde Y^2]}$, which is stated here without proof. It equals $+1$ or $-1$ exactly when $Y$ is, almost surely, an increasing or a decreasing linear function of $X$, and it is $0$ when the variables are uncorrelated. As an example let $X$ and $Y$ be two independent dice and $S = X + Y$ their sum. Then $\operatorname{Cov}(X, S) = \operatorname{Var} X + \operatorname{Cov}(X, Y) = 35/12$, because the second die is independent of the first, and $\rho = (35/12)/(1.708 \cdot 2.415) \approx 0.7071$, which is $1/\sqrt 2$: the first die supplies half of the variance of the sum, and $\rho^2 = 1/2$.

Uncorrelated variables that are not independent

Independence implies zero covariance, but the converse is false. Let $X$ be uniform on $\{-1, 0, 1\}$, each value with probability $1/3$, and let $Y = X^2$. Then $\mathbb{E}[X] = 0$, and since $X^3 = X$ for these values, $\mathbb{E}[XY] = \mathbb{E}[X^3] = \mathbb{E}[X] = 0$, so

Yet $Y$ is a function of $X$, so the pair is about as dependent as a pair can be. The product rule fails at once: $Y = 1$ happens with probability $2/3$ and $X = 0$ with probability $1/3$, but $X = 0$ forces $Y = 0$, so $\mathbb{P}(X = 0, Y = 1) = 0$, while $\mathbb{P}(X = 0)\,\mathbb{P}(Y = 1) = \tfrac13 \cdot \tfrac23 = 2/9$.

The explanation is that covariance only detects a linear tendency: whether larger $X$ goes with larger $Y$ on average. Here $Y$ falls as $X$ rises from $-1$ to $0$ and rises again from $0$ to $1$, and the two trends cancel exactly. Independence is a statement about all pairs of events at once, correlation is one number computed from them, and zero correlation never proves independence.

Independent coin tosses inside one uniform number

"Probability spaces and Kolmogorov's axioms" modelled infinitely many coin tosses by the binary digits of a number $\omega$ drawn uniformly from $[0, 1)$, written here $d_1(\omega), d_2(\omega), \ldots$, and treated them as separate fair coins without proving that they are independent. The tools are now in place. The first digit is $1$ exactly on $[\tfrac12, 1)$, so $\mathbb{P}(d_1 = 1) = \tfrac12$. The second digit is $1$ on $[\tfrac14, \tfrac12) \cup [\tfrac34, 1)$, again of total length $\tfrac12$. In general $d_k = 1$ on $2^{k-1}$ intervals of length $2^{-k}$, so every digit is a fair coin. Two digits together: $\{d_1 = 1, d_2 = 1\} = [\tfrac34, 1)$ has length $\tfrac14 = \tfrac12 \cdot \tfrac12$.

The general case is the same computation. Fix the first $k$ digits to prescribed values. They determine one dyadic interval $[m/2^k, (m+1)/2^k)$, where $m$ is the integer with those binary digits, of length $2^{-k}$, which is the product of $k$ factors $\tfrac12$. Digits at arbitrary positions $i_1 < \cdots < i_j$ are no harder. Let $n = i_j$ be the last position. The $n - j$ unconstrained digits among the first $n$ can take any of $2^{n-j}$ values, so the event is a union of $2^{n-j}$ dyadic intervals of length $2^{-n}$, with total length $2^{-j}$, which is again the product of $j$ factors $\tfrac12$. The product rule therefore holds for every finite set of digits: the digits $d_1, d_2, \ldots$ are mutually independent, and each is a fair coin.

One uniform number therefore holds an infinite sequence of independent fair coins. It holds much more. Deal the digits of $\omega$ into two sequences, the odd positions and the even positions, and read each as a binary number: $U = 0.d_1 d_3 d_5 \ldots_2$ and $V = 0.d_2 d_4 d_6 \ldots_2$. The first $k$ digits of $U$ are $k$ independent fair coins, so $U$ lands in each dyadic interval of length $2^{-k}$ with probability $2^{-k}$, and since dyadic intervals are closed under intersection and generate the Borel sets, $U$ is uniform on $[0, 1)$; likewise $V$. The two sequences are built from disjoint groups of independent coins, and functions of disjoint groups of independent variables are independent, by the same argument as for functions of two variables. So $U$ and $V$ are independent uniforms, and by the previous sections $(U, V)$ is a point drawn by area on the unit square. The map $\omega \mapsto (U, V)$ carries length on $[0, 1)$ onto area on the square. For $\omega = 0.8$ the first eight digits give $U = V = 0.625$ and the full expansions give $U = V = 2/3$.

Dealing the digits into countably many sequences, by the rows of a table, gives a whole sequence of independent uniform numbers from one. Applying the quantile function of "Random variables and their distributions" to each gives independent variables of any prescribed distributions. The interval $[0, 1)$ with length is thus a sample space large enough for any sequence of independent random variables. That infinite products of probability measures exist in general is Kolmogorov's extension theorem; the construction above is a concrete instance. The next lesson, "Modes of convergence and the law of large numbers", uses independent tosses to show how frequencies settle down.

Summary

• Events are independent when $\mathbb{P}(A \cap B) = \mathbb{P}(A)\mathbb{P}(B)$, equivalently when conditioning on one leaves the other unchanged; for several events the product rule must hold for every sub-collection, since independence in pairs is not enough.
• σ-algebras are independent when all their events are; random variables are independent when $\sigma(X)$ and $\sigma(Y)$ are, which can be checked on half-lines, and functions of independent variables stay independent.
• The product measure $\mu_1 \otimes \mu_2$ is the unique measure that gives every rectangle the product of the sides; area is length $\otimes$ length, and the joint distribution of independent variables is the product of their distributions.
• Fubini and Tonelli let you integrate over a product in either order, and give $\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y]$ for independent variables.
• $\operatorname{Var}(X + Y) = \operatorname{Var} X + \operatorname{Var} Y + 2\operatorname{Cov}(X, Y)$, so variances of independent variables add; uncorrelated does not imply independent.
• The binary digits of a uniform number are independent fair coins, and dealing them into two sequences produces two independent uniform numbers.