Total Variance Distance

Let $(\Omega, \mathcal{F})$ be a measurable space, $\mu$ and $\nu$ be two probability measure.

Definition. The total variance distance between $\mu$ and $\nu$ is defined as

$$\Vert \mu - \nu \Vert_{TV} \doteq \sup_{ E \in \mathcal{F} } \big| \mu(E) - \nu(E) \big|.$$

Property. $\Vert \cdot \Vert_{TV}$ is a metric.

Proof.

1. Clearly $\Vert \cdot \Vert_{TV}$ is non-negative.

2. If $\Vert \mu - \nu \Vert_{TV} = 0$, then $\mu(E) = \nu(E)$, for each $E \in \mathcal{F}$. Therefore, $\mu = \nu$.

3. Let $\kappa$ be another probability measure. Then

$\square$

Property.

$$\Vert \mu - \nu \Vert_{TV} \doteq \frac{1}{2} \cdot \sup \left\{ \int f \, d \mu - \int f \, d \nu : \Vert f \Vert_\infty \le 1 \right\}.$$

Proof. Since both $\mu$ and $\nu$ are probability measure, $\kappa \doteq \mu - \nu$ is a signed measure. Via Hahn Decomposition,

Hence, $\Vert \mu - \nu \Vert_{TV} = \mu(A) - \nu(A) = \kappa(A)$. Let $f \doteq \mathbf{1}_{A} - \mathbf{1}_{\bar A}$. Then $\Vert f \Vert_\infty \le 1$, and

$\square$