Let (Ω,F,μ)(\Omega, \mathcal{F}, \mu) be a probability space. Let X:Ω→RX: \Omega \rightarrow \mathbb{R} be a random variable, s.t., X−1(B)∈FX^{-1}(B) \in \mathcal{F}, ∀B∈R\forall B \in \mathcal{R}, where R\mathcal{R} is the collection of Borel sets in R\mathbb{R}.

Suppose that φ\varphi is a convex function defined on the interval I=[a,b]I = [a, b], and that X(Ω)⊂IX(\Omega) \subset I.

Assume that both XX and φ∘X\varphi \circ X are integrable.

Theorem. (Jensen's Inequality)

φ(E[X])=φ(∫X d μ)≤∫φ(X) d μ=E[φ(X)]. \varphi (\mathbb{E}[ X ]) = \varphi \left( \int X \, d\, \mu \right) \le \int \varphi (X) \, d\, \mu = \mathbb{E}[ \varphi (X) ].

Proof. Since X(Ω)⊂IX(\Omega) \subset I, by monotonicity of integration, it holds that

x∗≐E[X]=∫X d μ∈I. x^* \doteq \mathbb{E}[ X ] = \int X \, d\, \mu \in I.

Case 1. Suppose that x∗∈(a,b)x^* \in (a, b). As φ\varphi is convex, there exits a linear function h(x)=c(x−x∗)+φ(x∗)h(x) = c (x - x^*) + \varphi( x^*) such that

  • φ(x)≥h(x)\varphi (x) \ge h(x) for all x∈Ix \in I.

Hence, by monotonicity of integration,

E[φ(X)]=∫φ(X) d μ≥∫c(X−x∗)+φ(x∗) d μ=φ(x∗)=φ(E[X]). \mathbb{E}[ \varphi (X) ] = \int \varphi (X) \, d\, \mu \ge \int c (X - x^*) + \varphi( x^*) \, d\, \mu = \varphi (x^*) = \varphi (\mathbb{E}[ X ]).

Case 2. x∗=ax^* = a (or bb). Then

∫(X−a) d μ=0. \int \big( X - a \big) \, d\, \mu = 0.

It follows that X(ω)=aX(\omega) = a a.e. ω∈Ω\omega \in \Omega. Clearly

φ(∫X d μ)=φ(a)=∫φ(X) d μ. \varphi \left( \int X \, d\, \mu \right) = \varphi (a) = \int \varphi (X) \, d\, \mu.

□\square

Example. Let φ(x)=−ln⁡x,∀x>0\varphi(x) = - \ln x, \forall x > 0. Let x1,…,xn∈(0,∞)x_1, \ldots, x_n \in (0, \infty). Denote xˉ=(1/n)∑ixi\bar x = (1 / n) \sum_{i} x_i.

φ(xˉ)≤1n∑i(−ln⁡xi). \varphi( \bar x ) \le \frac{1}{n} \sum_i (- \ln x_i ).

Instead invoking the theorem directly, we apply the same technique to prove this again. The claim is trivial if x1=…=xnx_1 = \ldots = x_n. Otherwise, consider the tangent line of φ(x)\varphi(x) at xˉ\bar x:

h(x)=−1xˉ⋅(x−xˉ)−ln⁡xˉ. h(x) = - \frac{1}{ \bar x } \cdot (x - \bar x) - \ln \bar x.

It holds that φ(xi)≥h(xi)\varphi(x_i) \ge h(x_i), for all i∈[n]i \in [n]. Therefore,

1n∑ih(xi)≤1n∑i(−ln⁡xi). \frac{1}{n} \sum_{i} h(x_i) \le \frac{1}{n} \sum_i (- \ln x_i ).

On the other hand,

1n∑ih(xi)=−1xˉ⋅(1n∑ixi−xˉ)−ln⁡xˉ=−ln⁡xˉ. \frac{1}{n} \sum_{i} h(x_i) = - \frac{1}{ \bar x } \cdot \Big( \frac{1}{n} \sum_{i} x_i - \bar x \Big) - \ln \bar x = - \ln \bar x.

Reference

[1] 实变函数论，第四章，例五，周民强.