Random variables and their distributions

"Measurable functions" showed how a function on a space can be measured: when $f$ is measurable, the set $f^{-1}(B)$ is an event for every Borel set $B$, and the image measure records how much of the space is sent into $B$. "Probability spaces and Kolmogorov's axioms" then supplied the spaces on which this will happen: two dice, a dart thrown at a square, a uniform number $U$ in $[0, 1]$, an endless sequence of coin tosses. This lesson puts the two together. A random variable is a measurable function on a probability space, a score, a count, a waiting time, and its distribution is its image measure, a probability measure on the number line that says how the values are spread. The lesson explains why one increasing function, the distribution function, describes the whole distribution, how atoms and densities appear in it, which distributions turn up most often, and how the inverse of the distribution function turns uniform numbers into draws from any distribution at all.

Random variables

Let $(\Omega, \mathcal{F}, \mathbb{P})$ be a probability space. A random variable is a measurable function $X : \Omega \to \mathbb{R}$. By "Measurable functions" this means that $\{X \in B\} = X^{-1}(B)$ is an event for every Borel set $B$, and by the half-line test it is enough that $\{X \le x\}$ is an event for every real number $x$. The name is old and slightly misleading: $X$ is neither random nor a variable. It is a fixed function that attaches a number to every outcome, and the randomness lies in which outcome $\omega$ the experiment produces. Writing $\mathbb{P}(X \le 3)$ is shorthand for $\mathbb{P}(\{\omega : X(\omega) \le 3\})$.

Measurability is exactly what is needed to ask the questions one wants to ask. The number $\mathbb{P}(X \le x)$ exists only if $\{X \le x\}$ is an event. Take a die whose observer sees only the parity of the face, with the σ-algebra $\{\emptyset, \{1, 3, 5\}, \{2, 4, 6\}, \Omega\}$. For this observer the event "the face is at most 2" does not exist, since $\{1, 2\}$ is not in the σ-algebra, so the face itself cannot be a random variable on this space, while "face mod 2" can. A random variable has to be computable from the information the events stand for.

Four examples, each on a model from "Probability spaces and Kolmogorov's axioms", show how varied random variables are.

• The sum of two dice. On the 36 outcomes $(d_1, d_2)$, each with probability $1/36$, let $S = d_1 + d_2$. When every subset of $\Omega$ is an event, every function is measurable. The value 7 comes from six outcomes, $(1, 6), (2, 5), \ldots, (6, 1)$, so $\mathbb{P}(S = 7) = 6/36 = 1/6$, while the value 2 comes from only $(1, 1)$, so $\mathbb{P}(S = 2) = 1/36$.
• Functions of a uniform number. On $\Omega = [0, 1]$ with Lebesgue measure, $U(\omega) = \omega$ is a random variable, and so are $U^2$ and $-\ln(1 - U)$, which is $+\infty$ only at the single outcome $\omega = 1$ of probability 0. The event $\{U^2 \le 0.25\}$ is $[0, 0.5]$, so $\mathbb{P}(U^2 \le 0.25) = 0.5$. The event $\{-\ln(1 - U) \le 1\}$ is $[0, 1 - e^{-1}]$, so its probability is $1 - e^{-1} = 0.6321$.
• Heads among the first tosses. On the coin model, where the tosses are the binary digits of $\omega$, let $H_n$ count the digits equal to 1 among the first $n$. It depends on finitely many digits, so it is constant on dyadic intervals and measurable. Among the eight equally likely patterns of three tosses, three have exactly two heads, so $\mathbb{P}(H_3 = 2) = 3/8$. For ten tosses, $\mathbb{P}(H_{10} = 5) = 252/1024 = 0.2461$.
• The distance of a dart from the centre. For a dart uniform on the unit square, let $R$ be its distance from the centre. The set $\{R \le r\}$ is a disc of radius $r$ around the centre, and for $r \le \tfrac12$ the disc lies inside the square, so its area is $\pi r^2$. Hence $\mathbb{P}(R \le 0.25) = \pi/16 = 0.1963$ and $\mathbb{P}(R \le 0.5) = \pi/4 = 0.7854$.

New random variables come from old ones by the rules of "Measurable functions". If $X$ is a random variable and $g$ is a Borel function, such as a continuous or monotone function, then $g(X)$ is a random variable, because $\{g(X) \in B\} = \{X \in g^{-1}(B)\}$. Sums, products, maxima and limits of random variables are random variables, which is why $U^2$ and $-\ln(1 - U)$ above needed no separate check.

The distribution of a random variable

A random variable moves probability from $\Omega$ onto the number line. Think of one unit of sand spread over $\Omega$, with the probability of an event as the amount of sand in it, and let $X$ carry every grain from $\omega$ to the position $X(\omega)$. The distribution of $X$, also called its law, is the image measure of $\mathbb{P}$ under $X$, as defined in "Measurable functions":

It tells how much sand ends up in $B$. It is a probability measure on the Borel sets: the whole line receives $\mathbb{P}(\Omega) = 1$, and disjoint sets $B_k$ have disjoint preimages, so countable additivity of $\mathbb{P}$ carries over. For the sum of two dice the distribution is a collection of point masses, $\mathbb{P}_S = \sum_{s=2}^{12} \frac{6 - |s - 7|}{36}\, \delta_s$, in the notation of "Measuring the size of sets": masses $1/36, 2/36, \ldots, 6/36, \ldots, 1/36$ at $2, 3, \ldots, 12$. For $U^2$ on the uniform space, $\mathbb{P}_{U^2}([0, c]) = \sqrt{c}$.

The distribution forgets which outcome produced which value. It keeps only how the values are spread, and different random variables can have exactly the same distribution. On $[0, 1]$ the variables $U$ and $1 - U$ satisfy $\mathbb{P}(U \le 0.3) = 0.3$ and $\mathbb{P}(1 - U \le 0.3) = \mathbb{P}(U \ge 0.7) = 0.3$, and the same holds for every interval, so they have the same distribution. Yet they are different functions: they agree only at $\omega = \tfrac12$, and they differ by more than $\tfrac12$ exactly when $U < \tfrac14$ or $U > \tfrac34$, which has probability $\tfrac12$. For two dice, $14 - S = (7 - d_1) + (7 - d_2)$, and $7 - d$ is again a uniform face, so $S$ and $14 - S$ have the same distribution while agreeing only when $S = 7$.

Two random variables with the same distribution are called equal in distribution, written $X \overset{d}{=} Y$. This is much weaker than $X = Y$: the first says that the values are spread alike, the second that they agree outcome by outcome. Random variables on different probability spaces can be equal in distribution as well. That is why a model is usually specified through a distribution, as in "$X$ is exponential with mean 2", without saying what $\Omega$ is.

The distribution function

A distribution is a function on sets, a large object. One number for each real $x$ turns out to be enough. The distribution function of $X$ is

In sand terms, $F(x)$ is the amount of sand at or to the left of $x$. For the uniform $U$, $F(x) = x$ on $[0, 1]$, $0$ to the left and $1$ to the right. For $U^2$ it is $\sqrt{x}$ on $[0, 1]$. For a fair die it is the staircase $F(x) = \lfloor x \rfloor / 6$ on $[0, 6]$, which stays at $0$ before the first face and at $1$ from the last face on. Four properties hold for every random variable, and each one follows from a rule of "Measuring the size of sets".

• Non-decreasing. If $x \le y$ then $\{X \le x\} \subseteq \{X \le y\}$, so $F(x) \le F(y)$ by monotonicity.
• Limits 0 and 1. The events $\{X \le n\}$ increase to $\Omega$, so $F(n) \to 1$ by continuity from below. The events $\{X \le -n\}$ decrease to $\emptyset$, so $F(-n) \to 0$ by continuity from above, which is allowed because the total mass is finite.
• Right-continuous. The event $\{X \le x\}$ is the intersection of the decreasing events $\{X \le x + 1/n\}$, so $F(x + 1/n) \downarrow F(x)$, again by continuity from above.
• Left limits. The event $\{X < x\}$ is the increasing union of $\{X \le x - 1/n\}$, so the limit from the left is $F(x-) = \mathbb{P}(X < x)$. It can be smaller than $F(x)$, because the point $x$ itself belongs to $\{X \le x\}$. This is why $F$ is continuous from the right but not necessarily from the left.

On the die, $F(3) = 3/6 = \tfrac12$ while just to the left of 3, $F(2.999) = 2/6 = \tfrac13$: a jump of size $\tfrac16$ at 3, with the value at the top of the jump. The properties let one read off probabilities. Since $\{X \le b\}$ is the disjoint union of $\{X \le a\}$ and $\{a < X \le b\}$ for $a < b$,

The second formula says that the probability of a single value is the size of the jump of $F$ there. For the die, $\mathbb{P}(2 < X \le 5) = F(5) - F(2) = \tfrac56 - \tfrac26 = \tfrac12$ and $\mathbb{P}(X = 3) = \tfrac12 - \tfrac13 = \tfrac16$.

The function $F$ determines the whole distribution. Two distributions with the same $F$ agree on every interval $(a, b]$. By the uniqueness argument of "Lebesgue measure and null sets", two probability measures that agree on a family of sets closed under intersection agree on the σ-algebra generated by it, and the half-open intervals generate the Borel sets of "σ-algebras and Borel sets". So $F$ is a complete description of $\mathbb{P}_X$. The converse also holds: every function that is non-decreasing and right-continuous with limits $0$ and $1$ is the distribution function of some random variable. Section 7 constructs it.

Atoms, densities and mixed laws

A value $x$ with $\mathbb{P}(X = x) > 0$ is an atom of the distribution: a point carrying a positive amount of sand, which shows as a jump of $F$. This word has nothing to do with the atoms of a σ-algebra, the blocks of a partition. A distribution has at most countably many atoms, because at most $n$ values can have probability at least $1/n$, since the total is 1. A distribution with all its mass on atoms is discrete, and the die and the sum of two dice are examples. Its distribution function is a staircase.

At the other extreme, a distribution has a density $f$ if $\mathbb{P}(a < X \le b) = \int_a^b f(x)\,dx$ for all $a < b$, with $f \ge 0$ and total area $1$. Then $F(x) = \int_{-\infty}^{x} f(t)\,dt$ is continuous, and where $f$ is continuous, $F' = f$. The density is the thickness of the sand layer, like the rod with density $\rho$ in "Measuring the size of sets": the mass in a short interval of width $h$ near $x$ is about $f(x)\,h$. The exponential distribution with mean 1 has density $f(x) = e^{-x}$ for $x \ge 0$, so $F(x) = 1 - e^{-x}$ and $\mathbb{P}(1 < X \le 3) = e^{-1} - e^{-3} = 0.3181$.

A density value is not a probability. It can exceed 1: the uniform distribution on $[0, \tfrac12]$ has density 2 there. Every single value has probability $0$, because the area under a curve over one point is zero, and so $F$ has no jumps.

The two kinds can be mixed. Someone arrives at a traffic light. With probability $0.7$ it is green and the wait is $0$ minutes; otherwise the wait is exponential with mean 2 minutes. The wait $X$ has

The function $F$ jumps by $0.7$ at $0$, so $\mathbb{P}(X = 0) = 0.7$ and $0$ is the only atom, and then it rises smoothly with slope $0.15\,e^{-x/2}$ towards 1. In sand terms the distribution is $0.7\,\delta_0$ plus $0.3$ times an exponential layer. Most drivers do not wait at all, yet the waits of the others are spread over a continuum of values.

A fourth shape exists, and it is rare outside examples. Let $\xi_1, \xi_2, \ldots$ be the tosses of the coin model, $0$ for tails and $1$ for heads, and let $X = \sum_{k \ge 1} 2\xi_k / 3^k$. The base-3 digits of $X$ are $0$ or $2$, so $X$ always lies in the Cantor set $C$ of "Lebesgue measure and null sets". Its distribution function is the Cantor function: $F = \tfrac12$ on the whole removed middle third $[\tfrac13, \tfrac23]$, $F = \tfrac14$ on $[\tfrac19, \tfrac29]$, $F = \tfrac34$ on $[\tfrac79, \tfrac89]$, and constant on every removed interval. The removed intervals have total length 1, so $F$ is flat almost everywhere, yet it climbs from 0 to 1, and all of the climb happens on $C$, which has length 0. This $F$ is continuous, so there are no atoms, and there is no density either: a density $f$ would give $C$ the probability $\int_C f\,dx = 0$, since $C$ has length zero, but $\mathbb{P}(X \in C) = 1$. Such distributions are called singular.

A catalogue of common laws

A handful of distributions account for most of the cases one meets. Each comes from a short story, and recognising the story is the quickest way to know the distribution.

• Bernoulli($p$). One toss of a coin that shows heads with probability $p$: $\mathbb{P}(X = 1) = p$ and $\mathbb{P}(X = 0) = 1 - p$. This is the indicator $\mathbf{1}_A$ of an event $A$ with $\mathbb{P}(A) = p$.
• Binomial($n$, $p$). The number of heads in $n$ tosses of such a coin, the tosses not influencing each other: $\mathbb{P}(X = k) = \binom{n}{k} p^k (1 - p)^{n-k}$ for $k = 0, \ldots, n$. For a fair coin and $n = 10$, $\mathbb{P}(X = 5) = 252/1024 = 0.2461$.
• Geometric($p$). The number of tosses up to and including the first head: $\mathbb{P}(X = k) = (1 - p)^{k-1} p$ for $k = 1, 2, \ldots$. For a fair coin $\mathbb{P}(X = 3) = \tfrac18$, the pattern tails, tails, heads. On the coin model, $\{X > k\}$ means that the first $k$ binary digits of $\omega$ are all 0, which is the interval $[0, 2^{-k})$, so $\mathbb{P}(X > k) = 2^{-k}$.
• Poisson($\lambda$). The number of events in a unit of time when they occur at random at an average rate $\lambda$: $\mathbb{P}(X = k) = e^{-\lambda} \lambda^k / k!$ for $k = 0, 1, 2, \ldots$. For $\lambda = 2$, $\mathbb{P}(X = 2) = 2e^{-2} = 0.2707$ and $\mathbb{P}(X \le 1) = 3e^{-2} = 0.4060$.
• Uniform($a$, $b$). A number with all positions in $[a, b]$ equally likely: density $1/(b - a)$ on $[a, b]$ and $F(x) = (x - a)/(b - a)$ there. The number $U$ of "Probability spaces and Kolmogorov's axioms" is Uniform($0$, $1$).
• Exponential(rate $\theta$). A waiting time without memory: density $\theta e^{-\theta x}$ and $F(x) = 1 - e^{-\theta x}$ for $x \ge 0$. Its mean is $1/\theta$, so mean 2 means rate $\tfrac12$.
• Normal($\mu$, $\sigma^2$). The bell curve, centred at $\mu$ with spread $\sigma$: density $\dfrac{1}{\sigma\sqrt{2\pi}}\, e^{-(x - \mu)^2 / (2\sigma^2)}$. Its distribution function has no closed formula. For $\mu = 0$ and $\sigma = 1$, $\mathbb{P}(|X| \le 1) = 0.6827$ and $\mathbb{P}(|X| \le 1.96) = 0.95$. A later lesson of this track explains why sums of many small contributions look like this.

The Poisson distribution is a limit of binomial ones. Cut the unit of time into $n$ short pieces and suppose that each piece contains one event with probability $\lambda/n$, the pieces not influencing each other. The number of events is then binomial($n$, $\lambda/n$), and for a fixed $k$,

The second factor tends to 1, since it is a product of $k$ factors each close to 1; the third tends to $e^{-\lambda}$; the fourth tends to 1. For $\lambda = 2$ and $k = 2$ the binomial probabilities are $0.3020$ for $n = 10$, $0.2734$ for $n = 100$ and $0.2709$ for $n = 1{,}000$, approaching the Poisson value $0.2707$.

The exponential distribution has no memory. Its tail is $\mathbb{P}(X > x) = e^{-\theta x}$, so $\mathbb{P}(X > s + t) = e^{-\theta s}\, e^{-\theta t} = \mathbb{P}(X > s)\, \mathbb{P}(X > t)$. In words: among the waits longer than $s$, the share that last more than $t$ further is $\mathbb{P}(X > s + t) / \mathbb{P}(X > s) = \mathbb{P}(X > t)$, the same share as for a wait that has just begun. A wait that has already lasted $s$ minutes is no closer to its end. With mean 2, $s = 1$ and $t = 2$, $\mathbb{P}(X > 3) = 0.2231$, and indeed $\mathbb{P}(X > 1)\,\mathbb{P}(X > 2) = 0.6065 \cdot 0.3679 = 0.2231$.

Quantiles

Turning $F$ around answers a different question: below which value does $X$ stay with probability $u$? When $F$ is continuous and strictly increasing, there is exactly one such value, the inverse function $F^{-1}(u)$. For the exponential distribution with mean 1, $F^{-1}(u) = -\ln(1 - u)$, so the median, $u = \tfrac12$, is $\ln 2 = 0.6931$, and the quartiles $u = \tfrac14$ and $u = \tfrac34$ are $0.2877$ and $1.3863$.

Jumps and flat parts break this. On the die, no $x$ has $F(x) = 0.4$, because $F$ jumps from $\tfrac13$ to $\tfrac12$ at 3. For the Cantor variable, every $x$ in $[\tfrac13, \tfrac23]$ has $F(x) = \tfrac12$. The quantile function repairs both with the smallest suitable value:

In words: the first point at which $X$ has stayed below with probability at least $u$. The infimum is needed because the set of such $x$ has no smallest element in general, and it is a minimum here because $F$ is right-continuous: the set $\{x : F(x) \ge u\}$ contains its left end, so $F(F^{-1}(u)) \ge u$. For the die, $u = 0.4$ gives $\{x : F(x) \ge 0.4\} = [3, \infty)$ and so $F^{-1}(0.4) = 3$. In fact every $u \in (\tfrac13, \tfrac12]$ gives 3, and $F^{-1}(0.5) = 3$ while $F^{-1}(0.51) = 4$. For the Cantor variable $F^{-1}(\tfrac12) = \tfrac13$, and the quantile function skips the whole gap up to $\tfrac23$.

For the traffic light, $F^{-1}(u) = 0$ for every $u \le 0.7$, because the atom at 0 already makes $F(0) = 0.7 \ge u$. Above $0.7$, solving $0.7 + 0.3(1 - e^{-x/2}) = u$ gives $F^{-1}(u) = -2\ln\big((1 - u)/0.3\big)$, so $F^{-1}(0.85) = -2 \ln 0.5 = 2 \ln 2 = 1.3863$ minutes. The median is $F^{-1}(0.5) = 0$: more than half of the drivers do not wait.

The one fact about $F^{-1}$ that matters below is an equivalence for every $x$ and every $u \in (0, 1)$:

If $u \le F(x)$, then $x$ belongs to the set whose infimum defines $F^{-1}(u)$, so $F^{-1}(u) \le x$. If $F^{-1}(u) \le x$, then $F(x) \ge F(F^{-1}(u)) \ge u$, using that $F$ is non-decreasing and that $F(F^{-1}(u)) \ge u$. The equivalence shows that $F^{-1}$ is non-decreasing, hence measurable by "Measurable functions".

Inverse transform sampling

The quantile function also runs in the other direction. Let $U$ be uniform on $(0, 1)$, the number $\omega$ of the uniform model, and let $F$ be any distribution function. Then $X = F^{-1}(U)$ has distribution function $F$:

The first equality is the equivalence above, and the second holds because $\mathbb{P}(U \le c) = c$ for $c \in [0, 1]$. By section 3, $F$ then determines the distribution of $X$, and this also proves the converse promised there: every function with the four properties is a distribution function. In sand terms, the uniform interval $(0, 1)$ is cut into pieces and mapped onto the number line. A jump of size $q$ in $F$ collects a piece of $u$-values of length $q$ and sends all of them to one value, and a flat part of $F$ is skipped.

Examples show how it works. For the die, $F^{-1}(u) = \lceil 6u \rceil$, and each face receives a piece of length $\tfrac16$ of the $u$-axis. For the exponential distribution with mean 2, $F(x) = 1 - e^{-x/2}$, so $F^{-1}(u) = -2\ln(1 - u)$, and $u = 0.9$ gives $-2 \ln 0.1 = 4.605$ minutes. For the traffic light, $u \le 0.7$ gives a wait of $0$ and $u = 0.85$ gives $1.386$ minutes. A computer produces uniform numbers and nothing else, and this is how every other distribution is made from them.

The figure shows one uniform number at a time. Choose a distribution and move $u$ on the vertical axis: the guide runs across to $F$ and down to the value $F^{-1}(u)$, and the readouts show that value and the probability of the atom there. The histogram collects 1,000 draws $F^{-1}(U)$, each from its own uniform number, and the last two readouts compare their average with the exact mean. At the default, the exponential distribution and $u = 0.5$, the value is $F^{-1}(0.5) = \ln 2 = 0.693$, there is no atom, and the average of the 1,000 draws is $1.049$ against the exact mean $1$. Switch to the traffic light: at $u = 0.5$ the guide hits the jump, the value is $0$ and the atom is $0.7$; of the 1,000 draws, 674 are exactly $0$. On the die, $u = 0.5$ gives the value $3$ with atom $\tfrac16 = 0.167$.

If $F$ is continuous, the step also runs backwards: $F(X)$ is uniform on $(0, 1)$, because $\mathbb{P}(F(X) \le u) = \mathbb{P}(X \le F^{-1}(u)) = F(F^{-1}(u)) = u$. This probability integral transform fails when $F$ has jumps: for the traffic light, $F(X) = 0.7$ with probability $0.7$.

Means: a first look

The last two readouts of the figure compare an average of draws with an exact number. The average is easy to compute from draws: add them up and divide by 1,000. The exact number is what such averages settle on as the number of draws grows. For the die it is $(1 + 2 + \cdots + 6)/6 = 3.5$, and the 1,000 draws of the figure averaged $3.611$. For the traffic light it is $0.7 \cdot 0 + 0.3 \cdot 2 = 0.6$ minutes, since 70% of the drivers wait nothing and the others wait 2 minutes on average, and the draws averaged $0.635$. For the exponential distribution with mean 1 it is $1$, against $1.049$. The gap shrinks as the number of draws grows.

Making this precise needs an integral against $\mathbb{P}_X$, and this is the topic of the next lesson, "Expectation, variance and inequalities". It defines the mean as an integral over $\Omega$, shows how to compute it from the distribution function, a density or a mixed law, and measures how widely draws scatter around it.

Summary

• A random variable is a measurable function on a probability space; its distribution $\mathbb{P}_X(B) = \mathbb{P}(X \in B)$ is the image measure, a probability measure on the Borel sets that forgets which outcome gave which value.
• The distribution function $F(x) = \mathbb{P}(X \le x)$ is non-decreasing and right-continuous with limits $0$ and $1$, and it determines the distribution: $\mathbb{P}(a < X \le b) = F(b) - F(a)$ and $\mathbb{P}(X = x) = F(x) - F(x-)$.
• Jumps of $F$ are atoms and slopes are densities; a distribution can be discrete, have a density, be mixed like the traffic light, or, rarely, be continuous without a density like the Cantor function.
• The common distributions, Bernoulli, binomial, geometric, Poisson, uniform, exponential and normal, each come from a short story, and the Poisson distribution is the limit of binomial($n$, $\lambda/n$).
• The quantile function $F^{-1}(u) = \inf\{x : F(x) \ge u\}$ satisfies $F^{-1}(u) \le x \iff u \le F(x)$, so $F^{-1}(U)$ has distribution function $F$ for a uniform $U$: inverse transform sampling.