Conditional probability and conditional expectation

The lesson "Probability spaces and Kolmogorov's axioms" turned probability into a measure of total mass one on a space of outcomes, "Expectation, variance and inequalities" defined the expectation $\mathbb{E}[X]$ as the integral of a random variable against that measure, and "Independence and product measures" introduced the conditional probability $\mathbb{P}(A \mid B)$ in brief, as the tool that independence needed. All of these describe a state of knowledge in which nothing has been observed. This lesson asks what changes when part of the answer is revealed. It starts with conditioning on a single event, derives the multiplication rule and Bayes' rule, and then moves to the conditional expectation of a random variable given a partition and finally given a σ-algebra, which is the best prediction that the information at hand allows.

Conditional probability

Recall the two dice of "Independence and product measures", with outcomes $(a, b)$, $a$ on the first die and $b$ on the second. A sum of 8 has probability $5/36 \approx 0.1389$, and once you are told that the first die shows 3, only the six outcomes $(3, 1), \ldots, (3, 6)$ remain, one of which, $(3, 5)$, has sum 8, so the probability becomes $1/6 \approx 0.1667$. This lesson starts again from that example and develops conditioning in full.

The recipe behind this calculation is to shrink the space of outcomes to the event that was observed, and then rescale so that the total probability is 1 again. For events $A$ and $B$ with $\mathbb{P}(B) > 0$, the conditional probability of $A$ given $B$ is

In words: take the part of $A$ that lies inside $B$ and measure it against the size of $B$. The numerator discards everything outside $B$, and the division by $\mathbb{P}(B)$ makes $B$ itself have probability $\mathbb{P}(B \mid B) = 1$. In frequency terms: throw the dice many times, keep only the throws whose first die shows 3, and about $1/6$ of those have sum 8.

As "Independence and product measures" noted, the new probability $\mathbb{P}(\cdot \mid B)$ is itself a probability measure on the same events, and the check is short. Its values are non-negative, being ratios of non-negative numbers; it gives the sure event the value $\mathbb{P}(\Omega \mid B) = \mathbb{P}(B)/\mathbb{P}(B) = 1$; and for disjoint events $A_1, A_2, \ldots$ the pieces $A_i \cap B$ are disjoint too, so countable additivity of $\mathbb{P}$ gives

Every rule derived from the axioms in "Probability spaces and Kolmogorov's axioms" therefore holds for conditional probabilities as well. For instance $\mathbb{P}(A^c \mid B) = 1 - \mathbb{P}(A \mid B)$. Information can raise or lower a probability. The chance that the sum is at least 10 is $6/36 = 1/6$. Given that the first die shows 6, three of the six possible outcomes reach it and the probability is $1/2$; given that it shows 1, none does and the probability is 0.

Conditioning is not symmetric. Reverse the roles in the example above and ask for the probability that the first die shows 3 given that the sum is 8. Only five outcomes have sum 8, and exactly one of them has a 3 in front:

The numerators agree, since both describe the outcome $(3, 5)$, but the denominators differ: one universe has six outcomes and the other five. Confusing the two conditional probabilities is the most common mistake in the subject, and the section on Bayes' rule shows how costly it can be.

The formula needs $\mathbb{P}(B) > 0$; otherwise it reads $0/0$. This is more than a technicality: a number drawn uniformly from $[0, 1]$ equals any given value with probability zero, yet conditioning on information about it can make sense. The section on σ-algebras defines conditioning without dividing by the probability of the observation.

The multiplication rule and total probability

Multiplying the definition by $\mathbb{P}(B)$ gives the multiplication rule:

In words: the chance that both events happen is the chance that the first happens, times the chance that the second happens once the first has. Applying the rule twice extends it to chains,

Take an urn with 3 red and 2 blue balls and draw three balls without putting any back. The first is red with probability $3/5$. Given that, 2 red balls remain among 4, so the second is red with probability $2/4$; given both, 1 red ball remains among 3. The chance that all three are red is $\tfrac35 \cdot \tfrac24 \cdot \tfrac13 = \tfrac1{10}$, which agrees with counting: one of the 10 equally likely sets of three balls consists of the three red ones.

The second rule splits a probability into cases. A partition of $\Omega$ is a list of disjoint events $B_1, \ldots, B_m$ of positive probability that together cover $\Omega$, as in "σ-algebras and Borel sets". Any event $A$ falls apart into the disjoint pieces $A \cap B_1, \ldots, A \cap B_m$, and adding their probabilities, each written with the multiplication rule, gives the law of total probability:

In words: the probability of $A$ is a weighted average of its probabilities in the separate cases, and the weights are the probabilities of the cases. Split the two dice by the value of the first die. Each case has probability $1/6$. Given first $= 1$, a sum of 8 is impossible; given first $= 2, \ldots, 6$ the second die must show one particular number, so the conditional probability is $1/6$. Then

as it should be. A second example: in the same urn, with nothing known about the first draw, the second ball is red with probability $\tfrac35 \cdot \tfrac24 + \tfrac25 \cdot \tfrac34 = \tfrac35$, exactly the probability that the first ball is red. The idea of weighting case-by-case answers by the probabilities of the cases returns below, with expectations in place of probabilities.

Bayes' rule

The multiplication rule can be applied in two orders, $\mathbb{P}(A \cap B) = \mathbb{P}(A \mid B)\,\mathbb{P}(B) = \mathbb{P}(B \mid A)\,\mathbb{P}(A)$. Dividing by $\mathbb{P}(A)$ turns one conditional probability into the other. This is Bayes' rule:

The denominator can be expanded with the law of total probability. The reading is a change of beliefs: $\mathbb{P}(B)$ is the prior, the probability of $B$ before anything is observed, and $\mathbb{P}(B \mid A)$ is the posterior, its probability after $A$ has been observed.

Here is the standard example. A disease affects 1% of a population, so the prevalence is $0.01$. A test comes out positive for 99% of the ill people, a figure called the sensitivity, $\mathbb{P}(\text{positive} \mid \text{ill}) = 0.99$. It comes out negative for 95% of the healthy people, called the specificity, $\mathbb{P}(\text{negative} \mid \text{healthy}) = 0.95$, so it is positive for the other 5% of the healthy. A randomly chosen person tests positive. How likely is it that this person is ill? Many people answer something close to 99%. The multiplication rule gives the two ways of being positive:

Their sum is $\mathbb{P}(\text{pos}) = 0.0594$, by the law of total probability, and Bayes' rule gives

The same calculation with people instead of probabilities makes the result easier to believe. Take 10,000 people. Of them 100 are ill and 99 of those test positive. The other 9,900 are healthy and 5% of them, 495 people, test positive too. In total 594 people are positive, and only 99 of them are ill: $99/594 = 1/6$.

The intuition fails because of the base rate. The sensitivity answers $\mathbb{P}(\text{positive} \mid \text{ill})$, but the question asked was the reverse, $\mathbb{P}(\text{ill} \mid \text{positive})$, and these are different numbers. Few people are ill, so even a small false-alarm rate, applied to the many healthy people, produces more false alarms (495) than true detections (99). A strong test lifts a probability of 1% to 16.7%, not to 99%.

Posteriors can be updated again: if the person takes a second test and it is also positive, the posterior of the first test becomes the prior of the second. The two results are taken to be independent once the person's true health state is fixed. The worked example carries out both steps.

Conditional expectation given an event

The conditional measure $\mathbb{P}(\cdot \mid B)$ is a probability measure, so it has expectations like any other. The conditional expectation of a random variable $X$ given an event $B$ with $\mathbb{P}(B) > 0$ is the expectation of $X$ computed under $\mathbb{P}(\cdot \mid B)$:

In words: average $X$ over $B$ only, with the weights given by $\mathbb{P}$, and rescale by the size of $B$. The indicator $\mathbf{1}_B$ switches $X$ off outside $B$, so the numerator is the integral of $X$ over $B$, and the denominator turns it into an average.

Let $X$ be the face of one die and $B$ the event that the face is even. Then $\mathbb{E}[X\,\mathbf{1}_B] = (2 + 4 + 6)/6 = 2$, $\mathbb{P}(B) = 1/2$, and $\mathbb{E}[X \mid B] = 2 / (1/2) = 4$, which is simply the average of the three equally likely faces 2, 4 and 6. For the odd faces the same computation gives 3. With two dice, $\mathbb{E}[\text{first} \mid \text{sum} = 8] = (2 + 3 + 4 + 5 + 6)/5 = 4$, because the five outcomes with sum 8 are equally likely. Without information the first die averages $3.5$, so learning that the sum is 8, above the typical 7, moves the best guess up to $4$. For a fixed event $\mathbb{E}[X \mid B]$ is a single number; the next section lets the event depend on the outcome.

Conditional expectation given a partition

Imagine an observer who does not see the outcome $\omega$ but is told which block of a partition $B_1, \ldots, B_m$ it lies in. "σ-algebras and Borel sets" described such an observer by the σ-algebra $\mathcal{G}$ of all unions of blocks, whose atoms are the blocks themselves. What should this observer predict for a random variable $X$? Inside block $B_i$ the natural prediction is $\mathbb{E}[X \mid B_i]$. This prediction depends on $\omega$ only through the block, so it is a random variable, called the conditional expectation of $X$ given $\mathcal{G}$:

In words: flatten $X$ on every block to its average over that block. The result is constant on each atom of $\mathcal{G}$, and by "Measurable functions" that is exactly what it means for a function to be measurable with respect to a σ-algebra generated by a partition. So $\mathbb{E}[X \mid \mathcal{G}]$ is $\mathcal{G}$-measurable: it uses no information beyond the blocks.

Take the face $X$ of a die with four different observers. One sees only the parity, with blocks $\{1, 3, 5\}$ and $\{2, 4, 6\}$, and predicts $3$ on the first and $4$ on the second. One sees only whether the face is at most 2, with blocks $\{1, 2\}$ and $\{3, 4, 5, 6\}$, and predicts $1.5$ and $4.5$. One sees nothing, so the only block is $\Omega$ and the prediction is the constant $3.5$. One sees the face itself, all six blocks are single points, and the prediction is $X$. The more the observer knows, the closer the prediction follows $X$.

For a larger example, let $\mathcal{G}$ be generated by the sum of the two dice, which has 11 possible values and so 11 blocks. By symmetry, $\mathbb{E}[\text{first} \mid \text{sum}] = \text{sum}/2$: the outcomes $(a, b)$ and $(b, a)$ have the same sum and are equally likely, so within every block the two dice have the same average, and these two averages add up to the sum. For sum 7 the block has six outcomes and the average is $3.5$; for sum 8 it is $4$, for sum 12 it is $6$, and for sum 2 it is $1$.

The block averages, weighted by the probabilities of the blocks, add up to the plain expectation. This is the law of total expectation:

The first equality weights each block average by its probability, the second uses the definition of $\mathbb{E}[X \mid B_i]$, and the third adds up the pieces of $X$. The parity observer gives $\tfrac12 \cdot 3 + \tfrac12 \cdot 4 = 3.5$, the "at most 2" observer $\tfrac13 \cdot 1.5 + \tfrac23 \cdot 4.5 = 3.5$, and the dice-sum observer $\sum_s \mathbb{P}(\text{sum} = s)\, s/2 = 126/36 = 3.5$: each time the expectation of one die.

Conditional expectation given a σ-algebra

Partitions are not enough. Let $\omega$ be uniform on $[0, 1]$ and let $D = |\omega - \tfrac12|$ be its distance from the middle. An observer who is told $D = d$ knows that $\omega$ is one of the two points $\tfrac12 - d$ and $\tfrac12 + d$, but each event $\{D = d\}$ has probability zero, so $\mathbb{E}[X \mid B] = \mathbb{E}[X\,\mathbf{1}_B]/\mathbb{P}(B)$ reads $0/0$. Still, "what is the average of $\omega^2$ given $D = d$?" has an obvious answer: the two points are equally likely, so it is $\tfrac12\big((\tfrac12 - d)^2 + (\tfrac12 + d)^2\big) = \tfrac14 + d^2$, which is $0.26$ for $d = 0.1$ and $0.3125$ for $d = 0.25$. A definition is needed that produces such answers without dividing by a zero probability. The way out is to say what the answer must satisfy, instead of how to compute it.

Let $X$ be an integrable random variable and let $\mathcal{G} \subseteq \mathcal{F}$ be a sub-σ-algebra, the events the observer can decide. The conditional expectation $\mathbb{E}[X \mid \mathcal{G}]$ is a random variable $Z$ with two properties:

Property (i) says that $Z$ is built from the observer's information alone: every question about $Z$, such as "is $Z$ at most 0.3?", is one that $\mathcal{G}$ can answer. Property (ii) says that on every event the observer can name, $Z$ has the same total as $X$, so $Z$ carries no bias. It is $X$ with all the detail that $\mathcal{G}$ cannot see averaged away.

Block averages satisfy both properties. A block average is constant on each block, which is (i). For $G = B_j$, $\int_{B_j} Z \, d\mathbb{P} = \mathbb{E}[X \mid B_j]\,\mathbb{P}(B_j) = \mathbb{E}[X\,\mathbf{1}_{B_j}] = \int_{B_j} X \, d\mathbb{P}$, and every other $G \in \mathcal{G}$ is a union of blocks, over which both sides add up. The distance example passes as well. Take $G = \{D \le 0.1\} = [0.4, 0.6]$, which has probability $0.2$. On one side $\int_G \omega^2 \, d\omega = (0.6^3 - 0.4^3)/3 \approx 0.050667$. On the other, $Z = \tfrac14 + D^2$ and $D$ has density 2 on $[0, \tfrac12]$, so $\int_G Z \, d\mathbb{P} = 0.25 \cdot 0.2 + \int_0^{0.1} 2d^2 \, dd \approx 0.05 + 0.000667 = 0.050667$.

Is there such a $Z$, and is it the only one? Uniqueness has a short proof. Suppose $Z$ and $Z'$ both satisfy (i) and (ii). The event $G = \{Z > Z'\}$ belongs to $\mathcal{G}$, and by (ii) $\int_G (Z - Z') \, d\mathbb{P} = 0$. The integrand is positive on $G$, so $G$ is a null set; the same holds for $\{Z < Z'\}$, hence $Z = Z'$ almost surely. Conditional expectation is determined up to a null set, and equalities between conditional expectations are always meant almost surely. Existence for every integrable $X$ is the Radon–Nikodym theorem, which a later lesson of this track states. For a partition, the explicit formula above provides it.

The most common case is conditioning on a random variable $Y$. Then $\mathbb{E}[X \mid Y]$ means $\mathbb{E}[X \mid \sigma(Y)]$, where $\sigma(Y)$ is the σ-algebra of the events $\{Y \in B\}$ from "Measurable functions". A $\sigma(Y)$-measurable random variable is a function of $Y$, so $\mathbb{E}[X \mid Y] = g(Y)$ for some function $g$, and the value $g(y)$ is what one calls $\mathbb{E}[X \mid Y = y]$. For the dice, $g(s) = s/2$; for the distance example, $g(d) = \tfrac14 + d^2$.

A conditional expectation $\mathbb{E}[X \mid \mathcal{G}]$ is a random variable, not a number: it takes a value for each $\omega$, namely the best answer given what $\mathcal{G}$ has revealed about $\omega$. Only $\mathbb{E}[X \mid B]$, for a fixed event, is a number.

A concrete σ-algebra comes from the coin model of "Probability spaces and Kolmogorov's axioms", where a uniform $\omega \in [0, 1]$ is read as an infinite sequence of fair coin tosses, its binary digits. Knowing the first $k$ tosses means knowing which of the $2^k$ dyadic blocks $[j/2^k, (j+1)/2^k)$ contains $\omega$, and the σ-algebra $\mathcal{G}_k$ generated by these blocks is the observer after $k$ tosses. For $X = \omega^2$ and $k = 2$ the blocks are the four quarters of $[0, 1]$, and the block averages are $\tfrac1{48}, \tfrac7{48}, \tfrac{19}{48}, \tfrac{37}{48}$, that is $0.0208$, $0.1458$, $0.3958$ and $0.7708$. Their average is $\tfrac13 = \mathbb{E}[\omega^2]$. With no tosses known the prediction is the constant $\tfrac13$; after one toss it is $\tfrac1{12} \approx 0.0833$ on the left half and $\tfrac7{12} \approx 0.5833$ on the right half.

The figure shows this staircase. Choose a variable $X$, move the slider $k$ from 0 to 4 and watch: at $k = 0$ the staircase is one flat line at $\mathbb{E}[X]$, and every extra toss splits each block in two, so that it follows $X$ more closely. The first readout does not change with $k$, because the block averages always average to $\mathbb{E}[X]$. At the defaults, $X = \omega^2$ and $k = 2$, there are four blocks, $\mathbb{E}[X] = 0.3333$, and the error readout shows $0.006858$, which the next section explains. Switch on the comparison guess to see a second staircase.

Best prediction

Why average within a block, and not take another value? Because the average is the best prediction in a precise sense. Suppose the observer must predict $X$ with a $\mathcal{G}$-measurable random variable $Z$ with $\mathbb{E}[Z^2] < \infty$, and measure its quality by the mean squared error $\mathbb{E}[(X - Z)^2]$. Without information $Z$ must be a constant, the best constant is $\mathbb{E}[X]$, and the error is the variance of $X$ from "Expectation, variance and inequalities". The claim is that with information, $M = \mathbb{E}[X \mid \mathcal{G}]$ is the best of all.

The proof has two steps. First, $X - M$ is orthogonal to everything the observer can build: $\mathbb{E}[(X - M)\,Z] = 0$ for every $\mathcal{G}$-measurable $Z$ with finite second moment. For $Z = \mathbf{1}_G$ with $G \in \mathcal{G}$ this is exactly property (ii), because $\mathbb{E}[(X - M)\,\mathbf{1}_G] = \int_G X \, d\mathbb{P} - \int_G M \, d\mathbb{P} = 0$. Linearity extends it to finite sums $\sum a_i \mathbf{1}_{G_i}$, and a general $Z$ is a limit of such sums (an idea, not a complete proof). Second, write $X - Z = (X - M) + (M - Z)$. The difference $M - Z$ is $\mathcal{G}$-measurable, so the cross term vanishes by orthogonality, and the squares add:

The second term is never negative, and it is zero only if $Z = M$ almost surely. So $M$ is the best prediction, and every other prediction is worse by exactly the squared distance to $M$. This is Pythagoras' theorem for random variables: $X - M$ is perpendicular to everything $\mathcal{G}$ can build.

The figure's numbers illustrate it. For $X = \omega^2$ and four blocks, the best staircase has error $0.006858$. Take instead the guess that evaluates $X$ at the midpoint of each block, with values $\tfrac1{64}, \tfrac9{64}, \tfrac{25}{64}, \tfrac{49}{64}$. Each differs from the block average by $1/192 = 0.005208$, so the extra error is $0.005208^2 = 0.0000271$ and the guess has error $0.006885$, which is what the figure shows when the comparison guess is on. The guess is good, since squares are nearly linear on short blocks, but it cannot beat the block average.

More information can only help. If $\mathcal{G} \subseteq \mathcal{H}$, every $\mathcal{G}$-measurable prediction is also $\mathcal{H}$-measurable, so the minimum over the larger class is at most the minimum over the smaller one. For $X = \omega^2$ the error is $0.0889$ with 1 block (no tosses known), then $0.0264$, $0.0069$ and $0.0017$ with 2, 4 and 8 blocks. Each extra toss cuts the error by a factor of 3.4, 3.8 and then 4.0. Applying the Pythagoras identity to the constant prediction $Z = \mathbb{E}[X]$ also splits the variance in two parts:

The first part is what the information leaves unexplained, the second is what it explains. With four blocks, $0.0889 = 0.0069 + 0.0820$: the first two tosses explain about 92% of the variance of $\omega^2$.

Rules of conditional expectation

All the rules below follow from the two defining properties and have a simple block picture. They hold almost surely, since a conditional expectation is determined only up to a null set.

Linearity. $\mathbb{E}[aX + bY \mid \mathcal{G}] = a\,\mathbb{E}[X \mid \mathcal{G}] + b\,\mathbb{E}[Y \mid \mathcal{G}]$. Averaging over a block is a linear operation. For two dice, $\mathbb{E}[\text{first} + \text{second} \mid \text{sum}] = \text{sum}$, which is $\text{sum}/2 + \text{sum}/2$. Monotonicity goes with it: if $X \le Y$ then $\mathbb{E}[X \mid \mathcal{G}] \le \mathbb{E}[Y \mid \mathcal{G}]$, because averaging preserves order.

Taking out what is known. If $Z$ is $\mathcal{G}$-measurable, then

Inside a block, $Z$ is a constant, and a constant factors out of an average. For two dice, $Z = \text{sum}$ is known once the sum is given, so $\mathbb{E}[\text{first} \cdot \text{sum} \mid \text{sum}] = \text{sum}^2/2$; given sum 8 the value is $32$, which is $8$ times the block average $4$. With $X = 1$ the rule gives $\mathbb{E}[Z \mid \mathcal{G}] = Z$: what the observer knows does not need predicting.

The tower property. If $\mathcal{G} \subseteq \mathcal{H}$, then

In words: predicting from the finer information and then forgetting the details down to $\mathcal{G}$ is the same as predicting from $\mathcal{G}$ directly, so the coarser information wins. Each coarse block is a union of fine blocks, and its average is the probability-weighted average of their averages. For $X = \omega^2$, going from four blocks to two: the left half has fine averages $\tfrac1{48}$ and $\tfrac7{48}$, with mean $\tfrac1{12}$, and the right half has $\tfrac{19}{48}$ and $\tfrac{37}{48}$, with mean $\tfrac7{12}$, exactly the two-block averages found above. The trivial σ-algebra for $\mathcal{G}$ gives the law of total expectation.

Independence. If $X$ is independent of $\mathcal{G}$, in the sense of "Independence and product measures", then $\mathbb{E}[X \mid \mathcal{G}] = \mathbb{E}[X]$: information that has nothing to do with $X$ leaves the prediction unchanged. For two dice, $\mathbb{E}[\text{second} \mid \text{first}] = 3.5$. The figure shows a more surprising case. The values of $X = 4\omega(1 - \omega)$ at mirror points $\omega$ and $1 - \omega$ agree, so the two halves of $[0, 1]$ look alike and the first toss says nothing about $X$. Choose this $X$ and set $k = 1$: both block averages equal $2/3$, and the error stays at its no-information value $0.0889$.

Conditional Jensen. If $\varphi$ is convex, then $\mathbb{E}[\varphi(X) \mid \mathcal{G}] \ge \varphi(\mathbb{E}[X \mid \mathcal{G}])$, which is Jensen's inequality from "Expectation, variance and inequalities" applied inside every block. With $\varphi(x) = x^2$ the gap in a block is the variance of $X$ inside it: for $X = \omega^2$ on the block $[0, \tfrac12)$, $\mathbb{E}[X^2 \mid \text{block}] = 0.0125$, the square of the block average is $(1/12)^2 = 0.006944$, and the difference $1/180 \approx 0.005556$ is the variance of $\omega^2$ within that half.

Summary

• $\mathbb{P}(A \mid B) = \mathbb{P}(A \cap B)/\mathbb{P}(B)$ shrinks the space to $B$ and rescales; it is a probability measure, but $\mathbb{P}(A \mid B) \ne \mathbb{P}(B \mid A)$ in general.
• The multiplication rule, the law of total probability and Bayes' rule follow from the definition; with a base rate of 1%, a positive result from a 99% sensitive, 95% specific test means illness with probability only $1/6$.
• Given a partition, $\mathbb{E}[X \mid \mathcal{G}]$ replaces $X$ by its average on each block, and the block averages average back to $\mathbb{E}[X]$.
• Given a σ-algebra, $\mathbb{E}[X \mid \mathcal{G}]$ is the $\mathcal{G}$-measurable random variable with the same integral as $X$ over every event of $\mathcal{G}$; it is unique up to a null set.
• It is the best mean-squared prediction of $X$ from $\mathcal{G}$: any other is worse by the squared distance to it, and more information never increases the error.
• The rules: linearity, monotonicity, taking out what is known, the tower property, independence, conditional Jensen.