Random Variable

Random Variable

Let $(\Omega, \mathcal{F}, \mathbb{P})$ be a probability space, and let $\mathcal{B}$ be the collection of Borel sets in $\mathbb{R}$.

Definition. A random variable $Y: \Omega \rightarrow \mathbb{R}$ on $(\Omega, \mathcal{F}, \mathbb{P})$ is a function that is $\mathcal{F} / \mathcal{B}$ measurable, i.e.,

$$Y^{-1}(E) \doteq \{ \omega \in \Omega : Y(\omega) \in E \} \in \mathcal{F}, \quad \forall E \in \mathcal{B}.$$

It is not easy to verify this condition for each $E \in \mathcal{B}$ directly. It suffices to check one of the following equivalent conditions:

1. $Y^{-1}( (-\infty, a] ) \in \mathcal{F}, \forall a \in \mathbb{R}$.
2. $Y^{-1}( (-\infty, a) ) \in \mathcal{F}, \forall a \in \mathbb{R}$.
3. $Y^{-1}( [a, \infty) ) \in \mathcal{F}, \forall a \in \mathbb{R}$.
4. $Y^{-1}( (a, \infty) ) \in \mathcal{F}, \forall a \in \mathbb{R}$.

The correctness follows from the following theorem.

Theorem. If there exists a subset $\mathcal{S} \subseteq 2^{\mathbb{R}}$ such that 1) $\mathcal{S}$ generates $\mathcal{B}$, i.e., $\sigma(\mathcal{S}) = \mathcal{B}$; 2) for each $E \in \mathcal{S}$, $Y^{-1}(E) \in \mathcal{F}$, then $Y$ is measurable.

Proof. Let $\mathcal{L} \doteq \{ E \in \mathcal{B} : Y^{-1}(E) \in \mathcal{F} \}$. We claim that $\mathcal{L}$ it is a $\sigma$-field in $\mathbb{R}$, by showing that it is closed under complement and countable union.

Via that $\mathcal{F}$ is a $\sigma$-field,

1. If $Y^{-1}(E) \in \mathcal{F}$, then $Y^{-1}(\mathbb{R} - E) = \Omega - Y^{-1}(E) \in \mathcal{F}$.

2. If $Y^{-1}(E_n) \in \mathcal{F}$ for disjoint $E_n, n \in \mathbb{N}^+$, then

   $$Y^{-1}( \cup_{n \in \mathbb{N}^+} E_n ) = \cup_{n \in \mathbb{N}^+} Y^{-1}( E_n ) \in \mathcal{F}.$$

Finally, $\mathcal{S} \subseteq \mathcal{L}$, then $\sigma(\mathcal{S}) \subseteq \mathcal{L}$, the theorem follows.

$\square$

Distribution

Definition. The distribution $\mu: \mathcal{B} \rightarrow \mathbb{R}_{\ge 0}$ of $Y$ is a probability measure on $(\mathbb{R}, \mathcal{B})$ defined by

$$\mu(E) = \mathbb{P}[ Y \in E ] = \mathbb{P} \circ Y^{-1} (E), \quad \forall E \in \mathcal{B},$$

where $Y \in E$ is a shorthand for $Y^{-1}(E)$.

Proof. We verify that $\mu$ is a probability measure. Note that $\mu(\emptyset) = 0$, and $\mu(\mathbb{R}) = 1$. $\mu$ is countably additive: consider disjoint $E_1, E_2, \ldots \in \mathcal{B}$,

Hence, $(\mathbb{R}, \mathcal{B}, \mu)$ constitutes a probability space.

$\square$

Properties

Theorem. There can't be an uncountable collection of disjoint sets in $\mathcal{B}$ with positive $\mu$-measure.

Proof. Let $\mathcal{C} \subset \mathcal{B}$ be a disjoint collection of sets with positive $\mu$-measure. Consider $(0, 1]$. For each $k \in \mathbb{N}^+$, let

Then $|\mathcal{D}_{1, k}| \le k$, since $1 / k \cdot |\mathcal{D}_{1, k}| \le \sum_{E \in \mathcal{D}_{1, k} } \mu(x) \le 1$. Therefore the set $\mathcal{C}_0 \doteq \cup_{k \ge 1} \mathcal{D}_{1, k}$ is countable.

Similarly we can define $\mathcal{C}_n$, for each interval $(n, n + 1]$. If $E \in \mathcal{C}$, then there exists at least one $n \in \mathbb{Z}$, such that $E \in \mathcal{C}_n$. So $\mathcal{C} = \cup_{n \in \mathbb{Z} } \mathcal{C}_n$ is countable.

$\square$

Comment: what would be the value of $1 / k \cdot |\mathcal{D}_{1, k}|$ if $\mathcal{D}_{1, k}$ is uncountable? One possible way to avoid such multiplication is by applying the axiom of choice. Suppose that the set $\mathcal{D}_{1, k}$ contains more than $k$ elements, then by the axiom of choice we can iteratively pick at most $k$ element (without replacement) from the set. We can not find more item from the set, since the intersection of $(0,1]$ with each element has $\mu$-measure at least $1 / k$.

Expectation

Definition. Let $(\Omega, \mathcal{F}, \mathbb{P})$ be a probability space and $Y$ is a random variable defined on it. The expectation of $Y$ is defined by

$$\mathbb{E}[ Y ] \doteq \int Y \, d \mathbb{P} = \int_{ \Omega } Y(\omega) \, \mathbb{P} ( d \omega ).$$

For nonnegative $Y$, $\mathbb{E}[ Y ]$ is always defined. For general $Y$, $\mathbb{E}[ Y ]$ is defined if at least one of $\mathbb{E}[ X_+ ]$ or $\mathbb{E}[ X_- ]$ is finite. $Y$ is integrable if $\mathbb{E}[ |Y| ] < \infty$.

Composition

Theorem. Consider a function $f: \mathbb{R} \rightarrow \mathbb{R}$ is that is $\mathcal{B} / \mathcal{B}$ measurable. The composition of $f$ and $Y$, denoted by $f \circ Y$ or $f(Y)$, is a random variable.

Proof. For each Borel set $E \in \mathcal{B}$,

1. Its inverse image $f^{-1} (E)$ is a Borel set since $f$ is measurable $\mathcal{B} / \mathcal{B}$.
2. The inverse image of $f^{-1} (E)$ belongs to $\mathcal{F}$ since $Y$ is a random variable.

It concludes that $\big( f \circ Y \big)^{-1}(E) \in \mathcal{F}$.

$\square$

Expectation Under Composition

On the one hand, $f \circ Y$ is random variable on $(\Omega, \mathcal{F}, \mathbb{P})$. It expectation is given by

Let $\mu$ be the distribution of $Y$. Since $(\mathbb{R}, \mathcal{B}, \mu)$ constitutes a probability space, and $f$ is $\mathcal{B} / \mathcal{B}$ measurable, it is a random variable defined on $(\mathbb{R}, \mathcal{B}, \mu)$. By definition, its expectation is given by

It is therefore of interest whether these two definitions are consistent, i.e., whether two integrals are equal.

Theorem. If $f$ is nonnegative and measurable $\mathcal{B} / \mathcal{B}$, then

$$\mathbb{E}[ f(Y) ] = \int_{ \Omega } f( Y(\omega) ) \, \mathbb{P} ( d \omega ) = \int_{ \mathbb{R} } f(x) \, \mu (d x) = \mathbb{E}[ f ].$$

Further, $f$ (not necessarily nonnegative) is integrable with respect to $\mu = \mathbb{P} \circ Y^{-1}$ if and only if $f \circ Y = f(Y)$ is integrable with respect to $\mathbb{P}$, in which case the above equation holds.

Proof.

1. Suppose $f = \mathbb{1}_{A}$ is an indicator function for a measurable set, then

   $$\begin{aligned} \mathbb{E}[ f(Y) ] &= \int_{ \Omega } f( Y(\omega) ) \, \mathbb{P} ( d \omega ) = \int_{ \Omega } \mathbb{1}_{A} ( Y(\omega) ) \, \mathbb{P} ( d \omega ) = \mathbb{P} [ Y \in A],\, \\ \mathbb{E}[ f ] &= \int_{ \mathbb{R} } f(x) \, \mu (d x) = \int_{ \mathbb{R} } \mathbb{1}_{A} (x) \, \mu (d x) = \mu (A) = \mathbb{P} [ Y \in A]. \end{aligned}$$

   Therefore, $\mathbb{E}[ f(Y) ] = \mathbb{E}[ f ]$.

2. By linearity, this holds for nonnegative simple functions.

3. Suppose that $f$ is a non-negative function that is measurable $\mathcal{B} / \mathcal{B}$, then there is a sequence of non-negative simple functions $f_n \uparrow f$. Hence $f_n(Y) \uparrow f(Y)$. For each $n \in \mathbb{N}^+$, it holds that

   $$\mathbb{E}[ f_n(Y) ] = \mathbb{E}[ f_n ].$$

   The claim follows by monotone convergence theorem.

For general function $f$ that is measurable $\mathcal{B} / \mathcal{B}$, applying the above claim to $|f|$ establishes integrability. Decomposing $f$ into positive part and negative parts proves the claim.

$\square$

Example. Taking $f(x) = x, \forall x \in \mathbb{R}$, we see

$$\mathbb{E}[ Y ] = \int_{ \Omega } Y (\omega) \, \mathbb{P} (d \omega) = \int_{ \mathbb{R} } x \, \mu (d x).$$

Density

Let $(\Psi, \mathcal{M}, \lambda)$ be arbitrary measurable space.

Theorem. Let $\delta: \Psi \rightarrow \mathbb{R}_{\ge 0}$ be a function that is measurable $\mathcal{M} / \mathcal{B}$. Then we can define a measure by

$$\nu(E) \doteq \int_{E} \delta d \lambda, \quad \forall E \in \mathcal{M}.$$

$\nu$ is said to have density $\delta$ with respect to $\lambda$.

Note that $\nu(E)$ can be $\infty$.

Proof. Clearly $\nu(\emptyset) = 0$. Let $E_n \in \mathcal{M}, n \in \mathbb{N}^+$ be disjoint. Then via monotone convergence theorem,

$\square$

Theorem. Let $Y: \Psi \rightarrow \mathbb{R}_{\ge 0}$ be measurable $\mathcal{M} / \mathcal{B}$. Then

$$\int Y \, d \nu = \int Y \cdot \delta \, d \lambda.$$

Proof.

1. If $Y = \mathbb{1}_{A}$ for $A \in \mathcal{M}$, then

   $$\begin{aligned} \int Y \, d \nu = \int \mathbb{1}_{A} \, d \nu = \nu(A) = \int_A \delta \, d \lambda = \int \mathbb{1}_{A} \cdot \delta \, d \lambda. \end{aligned}$$

2. Via linearity of expectation, this holds for all non-negative simple functions.

3. Let $X_n, n \in \mathbb{N}^+$ be non-negative simple functions such that $X_n \uparrow Y$. Then $X_n \cdot \delta \uparrow Y \cdot \delta$. Via monotone convergence theorem, we get

   $$\int Y \, d \nu = \lim_n \int X_n \, d \nu = \lim_n \int X_n \cdot \delta \, d \lambda = \int Y \cdot \delta \, d \lambda.$$

$\square$

Corollary. Let $Y: \Psi \rightarrow \mathbb{R}$ be measurable $\mathcal{M} / \mathcal{B}$. It is integrable with respect to $\nu$ if and only if $Y \cdot f$ is integrable with respect to $\lambda$.

Proof.

By applying the previous theorem to $|Y|$.

$\square$

Equality

Theorem. If $Y$ is non-negative, then

$$\mathbb{E}[ Y ] = \int_{ \mathbb{R} } x \, d \mu = \int_{ \mathbb{R} } \mathbb{P}[ Y > t ] \, d t = \int_{ \mathbb{R} } \mathbb{P}[ Y \ge t ] \, d t,$$

where $\int_{ \mathbb{R} } \cdot \, d t$ is the Lebesgue integral.

Proof. Since there are countably many of points in $\mathbb{R}$ for which $\mathbb{P}(Y = x) \neq 0$, therefore the set for which $\mathbb{P}(Y \ge t) \neq \mathbb{P}(Y > t)$ has Lebesgue measure 0. It holds that

Case One: $Y$ is simple.

If $Y = \sum_{i \in [n] } a_i \mathbf{1}_{A_i}$ is a simple non-negative function such that the

are distinct, and the $A_i$ are disjoint. For each $i \in [n]$, define $\Delta_i = a_i - a_{i - 1}$. Then

Since for each $t \in (a_{i - 1}, a_i]$, it holds that $\mathbb{P}[Y \ge t] = \mathbb{P}[ Y \ge a_i]$, and for each $t > a_n$, it holds that $\mathbb{P}[Y \ge t] = 0$, we get

Since $\mathbb{P}(Y \ge t) = \mathbb{P} (Y > t)$ almost everywhere

Case Two: $Y$ is general non-negative measurable function.

If $Y$ is non-negative, then there is an increasing sequence of simple random variables $\{ X_n \} \uparrow Y$. Hence

As

By applying monotone convergence theorem to both hand side,

Remark: We can't argue that $\mathbb{P}(X_n \ge t) \uparrow \mathbb{P}(Y \ge t)$, even though $X_n \uparrow Y$. For example, the functions $X_n = (1 - 1 / n) \cdot \mathbb{1}_{[0, 1]}$ increases to $Y = \mathbb{1}_{[0, 1]}$. But $\mathbb{P}(X_n \ge 1) = 0$ and $\mathbb{P}(Y \ge 1) = 1$.

$\square$