Norm of Random Variables

Fix probability space $(\Omega, \mathcal{F}, \Pr)$ , for real number $p > 0$ , let $L_p$ denote the vector space of real value random variable for which $\mathbb{E} |X|^p < \infty$ .

For $p \ge 1$ , the $p$ norm $\Vert X \Vert_p$ of a random variable $X \in L_p$ is defined as

\Vert X \Vert_p = \left( \mathbb{E} \big[ |X|^p \big] \right)^{1 / p}.

Norm

$\Vert X \Vert_p$ is a norm.

Proof. This can be proven directly by Minkowski’s inequality. For completeness, we give the details here. Let $X, Y \in L_p$ , and let $a = \Vert X \Vert_p$ and $b = \Vert Y \Vert_p$ . We want to prove that

\Vert X + Y \Vert_p \le a + b \Longleftrightarrow \left\Vert \frac{a}{ a + b} \cdot \frac{X}{a} + \frac{a}{ a + b} \cdot \frac{Y}{a} \right\Vert_p \le 1.

It suffices to assume that $\Vert X \Vert_p = \Vert Y \Vert_p = 1$ and prove that

For $t \in [0, 1]$ , $\Vert t X + (1 - t) Y \Vert_p \le 1$ .

This is much easier to prove as we can take the power of $p$ of both side and show that:

\begin{aligned} \Vert t X + (1 - t) Y \Vert_p^p &= \mathbb{E}[ |t X + (1 - t) Y |^p ] \\ &\le \mathbb{E}[ | tX |^p + | (1 - t) Y|^p ] \\ &= \mathbb{E}[ | tX |^p ] + \mathbb{E} [ | ( 1 - t) Y|^p ] \\ &= t^p \Vert X \Vert_p^p + ( 1 - t)^p \Vert Y \Vert_p^p \\ &\le t + (1 - t) \\ &= 1. \end{aligned}

$\square$

Properties

Theorem. For $0 \le p \le q$ , $\big( \mathbb{E} [ |X|^p ] \big)^{1 / p} \le \big( \mathbb{E} [ |X|^q ] \big)^{1 / q}$ .

Corollary For $1 \le p \le q$ , $\Vert X \Vert_p \le \Vert X \Vert_q$ .

Proof. By convexity of the function $y = |x|^{q / p}$ , we have

\begin{aligned} \Big| \mathbb{E} \big[ |X|^p \big] \Big|^{q / p} \le \mathbb{E} \left[ \Big| |X|^p \Big|^{q / p} \right] = \mathbb{E} \left[ |X|^q \right] \end{aligned}

Raising both hand sides to the power of $1 / q$ finish the proof.

$\square$

Let $X_1, ..., X_n \in L_p$ be independent random variables and let $r_1, ... ,r_n$ be independent Rademacher random variables. Then
$\left\Vert \sum_{i \in [n] } X_i - \mathbb{E} \left[ \sum_{i \in [n] } X_i \right] \right\Vert_p \le 2 \cdot \left\Vert \sum_{i \in [n] } r_i X_i \right\Vert_p$

Proof. For $i \in [n]$ , let $Y_i$ be independent copy of $X_i$ . Then

\begin{aligned} \left\Vert \sum_{i \in [n] } X_i - \mathbb{E} \left[ \sum_{i \in [n] } X_i \right] \right\Vert_p &= \left\Vert \sum_{i \in [n] } X_i - \mathbb{E}_Y \left[ \sum_{i \in [n] } Y_i \right] \right\Vert_p \\ &= \left( \mathbb{E}_X \left[ \left| \sum_{i \in [n] } X_i - \mathbb{E}_Y \left[ \sum_{i \in [n] } Y_i \right] \right|^p \right] \right)^{1 / p} \\ &\le \left( \mathbb{E}_X \mathbb{E}_Y \left[ \left| \sum_{i \in [n] } X_i - \sum_{i \in [n] } Y_i \right|^p \right] \right)^{1 / p} &( \text{Jensen}) \\ &= \left\Vert \sum_{i \in [n] } (X_i - Y_i) \right\Vert_p \end{aligned}

Let $Z = (Z_1, ..., Z_n)$ be a sequence of independent random variables, such that either $Z_i = X_i - Y_i$ or $Z_i = Y_i - X_i$ . Let $\Gamma = \{ Z \}$ be the set of $2^n$ such sequences. As $X_i - Y_i$ and $Y_i - X_i$ have the same distributions, we have for $Z \in \Gamma$ ,

\mathbb{E}_Z \left[ \left| \sum_{i \in [n] } Z_i \right|^p \right] = \mathbb{E}_{X, Y} \left[ \left| \sum_{i \in [n]} X_i - Y_i \right|^p \right].

Therefore,

\frac{1}{ |\Gamma| } \sum_{ Z \in \Gamma } \mathbb{E}_Z \left[ \left| \sum_{i \in [n] } Z_i \right|^p \right] = \mathbb{E}_{X, Y} \left[ \left| \sum_{i \in [n]} X_i - Y_i \right|^p \right].

If $R = (r_1, ..., r_n)$ are Rademacher random variables, then $(r_1 (X_1 - Y_1), ..., r_n(X_n - Y_n)$ gives a uniformly sequence in $\Gamma$ . It holds that

\mathbb{E}_R \mathbb{E}_{X, Y} \left[ \left| \sum_{i \in [n]} r_i (X_i - Y_i) \right|^p \right] = \frac{1}{ |\Gamma| } \sum_{ Z \in \Gamma } \mathbb{E}_Z \left[ \left| \sum_{i \in [n] } Z_i \right|^p \right] = \mathbb{E}_{X, Y} \left[ \left| \sum_{i \in [n]} X_i - Y_i \right|^p \right].

Hence,

\begin{aligned} \left\Vert \sum_{i \in [n] } X_i - \mathbb{E} \left[ \sum_{i \in [n] } X_i \right] \right\Vert_p &\le \left\Vert \sum_{i \in [n] } (X_i - Y_i) \right\Vert_p \\ &= \left\Vert \sum_{i \in [n] } r_i(X_i - Y_i) \right\Vert_p \\ &\le 2 \left\Vert \sum_{i \in [n] } r_i X_i \right\Vert_p \end{aligned}

$\square$

Remark: We can simply the argument as follows. For a fixed sequence $(r_1,..., r_n) \in \{-1, 1 \}^n$ , the sequence $(r_1(X_1 - Y_1), ..., r_n(X_n - Y_n) )$ has the same distribution as $(X_1 - Y_n, ..., X_n - Y_n)$ . Averaging over all such possible sequences gives that

\left\Vert \sum_{i \in [n] } (X_i - Y_i) \right\Vert_p = \left\Vert \sum_{i \in [n] } r_i(X_i - Y_i) \right\Vert_p.

Let $X_1, ..., X_n$ be independent and mean zero random variables and let $Y_1, ..., Y_n$ be independent copies of them. Then for any $(a_{i, j} )$ and for $p \ge 1$ , it holds that
$\begin{aligned} \left\Vert \sum_{i \neq j} a_{i,j} X_i X_j \right\Vert_p \le 4\left\Vert \sum_{i \neq j} a_{i,j} X_i Y_j \right\Vert_p \end{aligned}$

Proof. Let $B_1, ..., B_n$ be independent Bernoulli random variables each of expectation $1 / 2$ . Then

\begin{aligned} \left\Vert \sum_{i \neq j} a_{i,j} X_i X_j \right\Vert_p &= \left( \mathbb{E}_X \left[ \left| \sum_{i \neq j} a_{i,j} X_i X_j \right|^p \right]\right)^{1 / p} \\ &= \left( \mathbb{E}_X \left[ \left| 4 \mathbb{E}_B \left[ \sum_{i \neq j} a_{i,j} X_i X_j B_i (1 - B_j) \right] \right|^p \right]\right)^{1 / p} \\ &\le \left( \mathbb{E}_X \left[ \mathbb{E}_B \left[ 4 \left| \sum_{i \neq j} a_{i,j} X_i X_j B_i (1 - B_j) \right|^p \right] \right]\right)^{1 / p} & (\text{Jensen} ) \\ &= 4 \left( \mathbb{E}_X \mathbb{E}_B \left[ \left| \sum_{i \neq j} a_{i,j} X_i X_j B_i (1 - B_j) \right|^p \right]\right)^{1 / p} \\ &= 4 \left( \mathbb{E}_B \mathbb{E}_X \left[ \left| \sum_{i \neq j} a_{i,j} X_i X_j B_i (1 - B_j) \right|^p \right]\right)^{1 / p} \end{aligned}

Question to ponder: Under what condition can we exchange the order of expectation (integration)?

But

\mathbb{E}_B \mathbb{E}_X \left[ \left| \sum_{i \neq j} a_{i,j} X_i X_j B_i (1 - B_j) \right|^p \right] \le \max_B \mathbb{E}_X \left[ \left| \sum_{i \neq j} a_{i,j} X_i X_j B_i (1 - B_j) \right|^p \right].

Let

B^* = (b^*_1, ..., b^*_n) = \arg \max_{B} \mathbb{E}_X \left[ \left| \sum_{i \neq j} a_{i,j} X_i X_j B_i (1 - B_j) \right|^p \right]

Define $S \doteq \{ i : b^*_i = 1 \}$ . Then

\begin{aligned} \mathbb{E}_X \left[ \left| \sum_{i \neq j} a_{i,j} X_i X_j b^*_i (1 - b^*_j) \right|^p \right] &= \mathbb{E}_X \left[ \left| \sum_{i \in S, j \notin S } a_{i,j} X_i X_j \right|^p \right] \\ &= \mathbb{E}_{X, Y} \left[ \left| \sum_{i \in S, j \notin S } a_{i,j} X_i Y_j \right|^p \right] \end{aligned}

Let $Z_1, ..., Z_n$ and $Z_1', ..., Z_n'$ both be independent copies of $X_1, ..., X_n$ . Then

\begin{aligned} \mathbb{E}_{X, Y} \left[ \left| \sum_{i \in S, j \notin S } a_{i,j} X_i Y_j \right|^p \right] &= \mathbb{E}_{X, Y} \left[ \left| \mathbb{E}_{Z, Z'} \left( \sum_{i \in S} a_{i,j} X_i \cdot \left( \sum_{j \notin S} Y_j + \sum_{j \in S} Z_j' \right) + \sum_{i \notin S} a_{i,j} Z_i \cdot \left( \sum_{j \notin S} Y_j + \sum_{j \in S} Z_j' \right) \right) \right|^p \right] & ( \mathbb{E}[Z_i ] = \mathbb{E} [ Z_j' ] = 0 ) \\ &\le \mathbb{E}_{X, Y} \mathbb{E}_{Z, Z'} \left[ \left| \sum_{i \in S} a_{i,j} X_i \cdot \left( \sum_{j \notin S} Y_j + \sum_{j \in S} Z_j' \right) + \sum_{i \notin S} a_{i,j} Z_i \cdot \left( \sum_{j \notin S} Y_j + \sum_{j \in S} Z_j' \right) \right|^p \right] & (\text{Jensen} ) \\ &=\mathbb{E}_{X, Y} \left[ \left| \sum_{i , j } a_{i,j} X_i Y_j \right|^p \right]. \end{aligned}

$\square$