Conditional Probability

Let $(\Omega, \mathcal{F}, \mathbb{P})$ be a probability space.

Definition (Conditional Probability of A Set). Given $A, B \in \mathcal{F}$, if $\mathbb{P}(B) \neq 0$, then the conditional probability of $A$ on $B$ is defined by

$$\mathbb{P}[ A \mid B ] \doteq \frac{\mathbb{P}[A \cap B]}{\mathbb{P}[B]}.$$

In general, the conditional probability is defined as a random variable measurable over a sub $\sigma$-field over $\mathcal{F}$. We present first the definition, and then use examples to illustrate it.

Definition (Conditional Probability). Let $\mathcal{G}$ be a $\sigma$-field in $\mathcal{F}$, and $A \in \mathcal{F}$. The conditional probability $\mathbb{P}[A \mid \mathcal{G}]$ is a random variable with two properties:

1. $\mathbb{P}[ A \mid \mathcal{G} ]$ is measurable $\mathcal{G}$ and integrable.
2. $\mathbb{P}[A \mid \mathcal{G}]$ satisfies the functional equation

$$\int_G \mathbb{P}[A \mid \mathcal{G}] \, d \mathbb{P} = \mathbb{P}[ A \cap G ], \quad \forall G \in \mathcal{G}.$$

The following provides the definition of conditional probability with respect to a random variable.

Definition. The conditional probability of $A \in \mathcal{F}$ given a random variable $X$ is defined as $\mathbb{P}[ A \mid \sigma(X) ]$ and is denoted $\mathbb{P}[A \mid X]$, where $\sigma(X)$ is the $\sigma$-field generated by $X$.

Remark. The $\sigma$-field $\sigma(X)$ consists of the sets $[\omega: X(\omega) \in H]$ for $H \in \mathcal{R}^1$.

Remark. From the experiment corresponding to the $\sigma$-field $\sigma(X)$, one learns which of the set $[\omega' : X(\omega') = x]$ contains $\omega$ and hence learns the value of $X(\omega)$.

Existence

It is easy to observe that if such a random variable exists and we alter its value on a set in $\mathcal{G}$ of measure 0, the conditions will still hold. The key challenge, however, is to prove the existence of such a random variable.

Theorem. $\mathbb{P}[A \mid \mathcal{G}]$ exists.

Proof. Define a measure $\mu_A$ in $\mathcal{G}$ by

Then $\mu_A$ is absolutely continuous w.r.t. $\mu$. The Randon-Nikodym theorem implies that there exists a function $f: \Omega \rightarrow \mathbb{R}$, measurable $\mathcal{G}$ and integrable w.r.t. $\mathbb{P}$, s.t.,

$\square$

Examples

Example. Let $A \in \mathcal{F}$. Let $\mathcal{G} = \lbrace \varnothing, \Omega \rbrace$. Define the random variable $X$ by

It is clear that $X(\omega)$ satisfies the two conditions required for the definition of conditional probability.

$\square$

Example. Let $A, B \in \mathcal{F}$ satisfies $\mathbb{P}[B] \neq 0$ and $\mathbb{P}[\bar{B}] \neq 0$. Let $\mathcal{G} = \lbrace \varnothing, B, \bar{B}, \Omega \rbrace$. Define the random variable $X$ by

Clearly, $X(\omega)$ satisfies the two conditions required for the definition of conditional probability.

$\square$

Example. Let $A \in \mathcal{F}$, and $B_n \in \mathcal{F}, n \in \mathbb{N}^+$ be a partition of $\Omega$ satisfying $\mathbb{P}[B_n] \neq 0$. Further, let $\mathcal{G}$ be the $\sigma$-field generated by the $B_n$. Define the random variable $X$ by

Once again, $X(\omega)$ satisfies the two conditions required for the definition of conditional probability.

$\square$

In the examples above, when interpreting the conditional probability as a function, we don't need to know the exact value of $\omega$ to determine the function output; knowing which set in the $\sigma$-field contains $\omega$ is sufficient. Requiring that $X(\omega) = \mathbb{P}[A \mid \mathcal{G}]_{\omega}$ to be measurable $\mathcal{G}$ essentially means that, if we can determine whether $\omega$ belongs to each set $H \in \mathcal{G}$, we can determine the function's output. Specifically, knowing which set $X^{-1}(a), \forall a \in \mathbb{R}$ contains $\omega$ allows us to identify the value of $X$.

Example. Assume that $\Omega = \mathbb{R}^2$, $\mathcal{F} = \mathcal{R}^2$ be the class of Borel sets, and $\mathbb{P}$ be a probability measure on with density $f(x, y)$ w.r.t the Lebesgue measure. Let $X(x, y) = x$. Further,

is the $\sigma$-field generated by the vertical strips : $\lbrace E \times \mathbb{R} = [ (x, y): x \in E ] : E \in \mathcal{R}^1 \rbrace$. Let $A = \mathbb{R} \times F = [(x, y ): y \in F]$, where $F \in \mathcal{R}^1$ be a horizontal strip. Then we claim that

We need to verify that

1. $\varphi(x, y)$ is measurable $\mathcal{G}$: first, note that both $\int_F f(x, t) \, dt$ and $\int_\mathbb{R} f(x, t) \, dt$ are measurable $\mathcal{R}$. So is their ratio. It follows that, for each $H \in \mathcal{R}$, $\varphi^{-1}(H) = E \times \mathbb{R}$ $E \in \mathcal{R}$.

2. For each $G = E \times \mathbb{R} \in \mathcal{G}$, where $E \in \mathcal{R}$, it holds that

   $$\begin{aligned} \int_{E \times \mathbb{R}} \varphi(x, y) \, d \mathbb{P} &= \int_{E \times \mathbb{R}} \varphi(x, y) \cdot f(x, y) \, d (x \times y) \\ &= \int_E \int_\mathbb{R} \frac{ \int_F f(x, t) \, dt }{ \int_\mathbb{R} f(x, t) \, dt } \cdot f(x, y) \, dy \, d x \\ &= \int_E \int_F f(x, t) \, dt \, d x \\ &= \mathbb{P}[E \times F] = \mathbb{P}[A \times G]. \end{aligned}$$

Example. Let $X : \Omega \rightarrow \mathbb{R}^j$ and $Y : \Omega \rightarrow \mathbb{R}^k$ be random vectors, with distributions $\mu$ and $\nu$ respectively. Suppose that $X$ and $Y$ are independent, then by Fubini's theorem, for each $H \in \mathcal{R}^j, J \in \mathcal{R}^{j + k}$,

We can construct the conditional probability $\mathbb{P}[(X, Y) \in J]_\omega$ as follows. Let $f(x) \doteq \mathbb{P}[ (x, Y) \in J] = \mathbb{P}[ \lbrace \omega : (x, Y(\omega)) \in J \rbrace ]$. Then we claim that $f(X(\omega))$ is a version of $\mathbb{P}[(X, Y) \in J \mid X ]_\omega$.

1. First, by Fubini's theorem, $f$ is measurable $\mathcal{R}$, therefore $f \circ X$ is measurable $\sigma(X)$.
2. Second, each $G \in \sigma(X)$ has the form $G = \left\lbrace \omega : X(\omega) \in H \right\rbrace$ for some $H \in \mathcal{R}$. Therefore,$$\begin{aligned} \mathbb{P}\left[ \left\lbrace \omega : X(\omega) \in H \right\rbrace \bigcap \left\lbrace \omega : (X(\omega), Y(\omega)) \in J \right\rbrace \right] &= \mathbb{P}[ X \in H, (X, Y) \in J ] \\ &= \int_{H} \mathbb{P}[ (x, Y) \in J] \, \mu(dx) \\ &= \int_{H} f(X) \, d \mathbb{P}. \end{aligned}$$

Properties of Conditional Probability

1. With probability $1$, $\mathbb{P}[\varnothing \mid \mathcal{G}] = 0$, and $\mathbb{P}[\Omega \mid \mathcal{G}] = 1$; and

   $$0 \le \mathbb{P}[A \mid \mathcal{G}] \le 1,$$

   for each $A \in \mathcal{G}$.

   Proof. Since $\mathbb{P}[\varnothing \mid \mathcal{G}]$ is measurable $\mathcal{G}$, $G_k \doteq \lbrace \omega : \mathbb{P}[\varnothing \mid \mathcal{G}]_\omega > \frac{1}{k} \rbrace \in \mathcal{G}$ for each $k \in \mathbb{N}^+$. It follows that

   $$\begin{aligned} 0 = \mathbb{P}[\varnothing \cap G_k] = \int_{G_k} \mathbb{P}[\varnothing \mid \mathcal{G}] \, d \mathbb{P} \ge \frac{1}{k} \cdot \mathbb{P}[G_k] = 0. \end{aligned}$$

   Therefore, the set $\{ \omega : \mathbb{P}[\varnothing \mid \mathcal{G}]_\omega > 0\} = \bigcup_{k \ge 1} G_k$ has measure $0$. By analogous argument, we can prove that for the random variable $- \mathbb{P}[\varnothing \mid \mathcal{G}]$ that the set $\{ \omega : - \mathbb{P}[\varnothing \mid \mathcal{G}]_\omega > 0\}$ has measure $0$.

   Similarly, we can show that $1 - \mathbb{P}[\Omega \mid \mathcal{G}] = 0$ with probability $1$.

   To prove the last claim, let $G_k \doteq \lbrace \omega : \mathbb{P}[\varnothing \mid \mathcal{G}]_\omega < -\frac{1}{k} \rbrace \in \mathcal{G}$ for each $k \in \mathbb{N}^+$. Then:

   $$\begin{aligned} 0 \le \mathbb{P}[A \cap G_k] = \int_{G_k} \mathbb{P}[A \mid \mathcal{G}] \, d \mathbb{P} < -\frac{1}{k} \cdot \mathbb{P}[G_k] = 0. \end{aligned}$$

   Thus, $\mathbb{P}[G_k] = 0$. Consequently, the set $\{ \omega : \mathbb{P}[A \mid \mathcal{G}]_\omega < 0\} = \bigcup_{k \ge 1} G_k$ has measure $0$. Applying the same reasoning to $1 - \mathbb{P}[A \mid \mathcal{G}]$ proves that $\mathbb{P}[A \mid \mathcal{G}] \le 1$ with probability $1$.

   $\square$

2. If $A_1, A_2, \ldots$ is a finite or countable sequence of disjoint sets, then

   $$\mathbb{P}[ \bigcup_{n \ge 1} A_n \mid \mathcal{G} ] = \sum_{n \ge 1} \mathbb{P}[A_n \mid \mathcal{G}].$$

   Proof. Without loss of generality, assume that $\mathbb{P}[A_n \mid \mathcal{G}] \ge 0$. For each $G \in \mathcal{G}$, it holds that

   $$\begin{aligned} \int_G \left( \mathbb{P}\left[ \bigcup_{n \ge 1} A_n \mid \mathcal{G} \right] - \sum_{n \ge 1} \mathbb{P}\left[A_n \mid \mathcal{G}\right] \right) \, d \mathbb{P} &= \int_G \mathbb{P}\left[ \bigcup_{n \ge 1} A_n \mid \mathcal{G} \right] \, d \mathbb{P} - \int_G \sum_{n \ge 1} \mathbb{P}\left[A_n \mid \mathcal{G}\right] \, d \mathbb{P} \\ &= \int_G \mathbb{P}\left[ \bigcup_{n \ge 1} A_n \mid \mathcal{G} \right] \, d \mathbb{P} - \sum_{n \ge 1} \int_G \mathbb{P}\left[A_n \mid \mathcal{G}\right] \, d \mathbb{P} \\ &= \mathbb{P}\left[ \bigcup_{n \ge 1} A_n \cap G \right] - \sum_{n \ge 1} \mathbb{P}[ A_n \cap G ] = 0, \end{aligned}$$

   where the second equality follows from monotone convergence theorem. By the previous property, we conclude that $\mathbb{P}[ \bigcup_{n \ge 1} A_n \mid \mathcal{G} ] - \sum_{n \ge 1} \mathbb{P}[A_n \mid \mathcal{G}] = 0$ with probability $1$.

   $\square$

3. If $A \subseteq B$, then

   $$\mathbb{P}[B - A \mid \mathcal{G}] = \mathbb{P}[B \mid \mathcal{G}] - \mathbb{P}[A \mid \mathcal{G}], \quad \mathbb{P}[A \mid \mathcal{G}] \le \mathbb{P}[B \mid \mathcal{G}].$$

   Proof. Based on the previous property,

   $$\mathbb{P}[B - A \mid \mathcal{G}] + \mathbb{P}[A \mid \mathcal{G}] = \mathbb{P}[B \mid \mathcal{G}],$$

   which finishes the proof.

   $\square$

4. If $A_n \uparrow A$, then

   $$\mathbb{P}[A_n \mid \mathcal{G}] \uparrow \mathbb{P}[A \mid \mathcal{G}].$$

   Proof. Define $B_1 = A_1$, and for each $n \ge 2$, let $B_n = A_n - A_{n - 1}$. Then the $B_n$ are mutually disjoint, and $\bigcup_{n \ge 1} B_n = A$. Based on property 3, it holds with probability $1$ that

   $$\mathbb{P}[A_1 \mid \mathcal{G}] \le \mathbb{P}[A_2 \mid \mathcal{G}] \le \cdots \le \mathbb{P}[A \mid \mathcal{G}].$$

   Further, based on property 2, it holds with probability $1$ that

   $$\begin{aligned} \mathbb{P}\left[ A_n \middle| \mathcal{G} \right] = \mathbb{P}\left[ \bigcup_{i = 1}^n B_n \middle| \mathcal{G} \right] = \sum_{i = 1}^n \mathbb{P}[B_n \mid \mathcal{G}], \\ \mathbb{P}\left[ A \middle| \mathcal{G} \right] = \mathbb{P}\left[ \bigcup_{n \ge 1} B_n \mid \mathcal{G} \right] = \sum_{n \ge 1} \mathbb{P}[B_n \mid \mathcal{G}]. \end{aligned}$$

   Therefore,

   $$\mathbb{P}\left[ A \middle| \mathcal{G} \right] = \lim_{n \rightarrow \infty} \sum_{i \in [n]} \mathbb{P}[B_n \mid \mathcal{G}] = \lim_{n \rightarrow \infty} \mathbb{P}\left[ A_n \middle| \mathcal{G} \right].$$

   $\square$

5. If $A_n \downarrow A$, then

   $$\mathbb{P}[A_n \mid \mathcal{G}] \, \Big\downarrow \, \mathbb{P}[A \mid \mathcal{G}].$$

   Proof. Let $B_n = A_1 - A_n$, and $B = A_1 - A$. Then $B_n \uparrow B$, based on property 4 we know that

   $$\mathbb{P}[A_1 - A_n \mid \mathcal{G}] \, \Big\uparrow \, \mathbb{P}[A_1 - A \mid \mathcal{G}].$$

   On the other hand, we know that

   $$\begin{aligned} \mathbb{P}[A_1 - A_n \mid \mathcal{G}] &= \mathbb{P}[A_1 \mid \mathcal{G}] - \mathbb{P}[A_n \mid \mathcal{G}], \\ \mathbb{P}[A_1 - A \mid \mathcal{G}] &= \mathbb{P}[A_1 \mid \mathcal{G}] - \mathbb{P}[A \mid \mathcal{G}]. \end{aligned}$$

   Noting that $\mathbb{P}[A_1 \mid \mathcal{G}] \le 1$ with probability $1$ completes the proof.

   $\square$

6. $\mathbb{P}(A) = 1 \implies \mathbb{P}[A \mid \mathcal{G}] = 1$.

   Proof. The proof is similar to the proof in property $1$, in which the essential property we are using are $\mathbb{P}[\varnothing] = 0$, and $\mathbb{P}[\Omega] = 1$.

   $\square$

7. $\mathbb{P}(A) = 0 \implies \mathbb{P}[A \mid \mathcal{G}] = 0$.

   Proof. Similar as property 6.

   $\square$

Conditional Probability distribution

Let $X$ be a random variable on $(\Omega, \mathcal{F}, \mathbb{P})$, and let $\mathcal{G}$ be a $\sigma$-field in $\mathcal{F}$.

Theorem. There exists a function $\mu(H, \omega)$, defined on $\mathcal{R} \times \Omega$, with these two properties:

1. For each $\omega \in \Omega$, $\mu(\cdot, \mu)$ is a probability measurable on $\mathcal{R}^1$.
2. For each $H \in \mathcal{R}$, $\mu(H, \cdot)$ is a version of $\mathbb{P}[X \in H \mid \mathcal{G}]$.

Proof. Define $F(r, \omega) = \mathbb{P}[X \le r \mid \mathcal{G}]_\omega$, for each $r \in \mathbb{Q}$. If $r \le s$, then based on the properties of conditional probability, with probability $1$:

As there are only countably many of these events, their intersection, denoted by $E$, has probability $1$.

• For each $\omega \in E$, extend $F(\cdot , \omega)$ to all of $\mathbb{R}$ by $F(x, \omega) \doteq \inf[ F(r, \omega) : x < r ]$. It is easy to see that $F(\cdot, \omega)$ is non-decreasing, right-continuous and therefore a probability distribution function. Therefore, there exists a probability measure $\mu(\cdot, \omega)$ on $(\mathbb{R}, \mathcal{R})$ with distribution function $F(\cdot, \omega)$.

• For each $\omega \notin E$, take $F(x, \omega) = F(x)$ for some arbitrary but fixed distribution function $F$, and let $\mu(\cdot, \omega)$ be the probability measure corresponding to $F(x)$.

Thus, condition 1 is satisfied.

Next, consider the class of $H$ for which $\mu(H, \cdot)$ is measurable $\mathcal{G}$:

Then

1. By definition of $F(r, \omega)$, $\mathcal{S}$ contains $(-\infty, r]$ for rational $r$.

2. $\mathcal{S}$ is a $\lambda$-system, since

   • $\Omega \in \mathcal{S}$.

   • Let $A, B \in cS$. Since $\mu$ is a measure, $\mu(A - B, \cdot) = \mu(A, \cdot) - \mu(B, \cdot)$. Since $\mu(A, \cdot)$ and $\mu(B, \cdot)$ are $\mathcal{G}$ measurable, so is $\mu(A - B, \cdot)$.

   • Let $A_n \in \mathcal{S}$, $n \ge 1$ are mutually disjoint. Since $\mu$ is a measure, $\mu(\bigcup_n A_n , \cdot) = \sum_{n \ge 1} \mu(A_n, \cdot)$. Since $\mu(A_n, \cdot)$ are $\mathcal{G}$ measurable, so is $\mu(\bigcup_{n \ge 1} A_n, \cdot)$.

Therefore, $\mathcal{S} = \mathcal{G}$. Finally, for each $r \in \mathbb{Q}$ and $H = (-\infty, r]$, with probability $1$, $\mu(H, \omega) = \mathbb{P}[X \le r \mid \mathcal{G}]_\omega$; Therefore, for each $G \in \mathcal{G}$,

Since the $H = (-\infty, r], r \in \mathbb{Q}$ is a $\pi$ system which generates $\mathcal{R}$, this holds for all $H \in \mathcal{R}$ as well.

$\square$

Reference

1. Billingsley, Patrick. "Probability and Measure." 3rd ed. Wiley Series in Probability and Mathematical Statistics. New York: Wiley, 1995.