Beta Distribution

We start with a possibly biased coin which lands on head with unknown probability $p$ when tossed. Suppose we obtain $Y = a$ heads and $Z = b$ tails out of $a + b$ tosses. Conditioned on $p = x$ for some $x \in (0, 1)$, we have

Assume further that we have equal prior of $p$ over $[0, 1]$, then the posterior probability density function $f(x)$ of $p$ satisfies

This motivates the definition of Beta distribution.

Definition (Beta distribution). A random variable $X$ follows the Beta distribution with parameters $a > 0$ and $b > 0$, written as $X \sim \mathcal{B}(a, b)$, if its density is

$$f(x) = \frac{1}{\beta(a, b)} \cdot x^{a−1} (1 − x)^{b−1}, \qquad 0 < x < 1,$$

where $\beta(a, b)$ is a constant.

Note that it is just a convention that we define $f(x) \propto x^{a - 1} (1 - x)^{b - 1}$ for $\mathcal{B}(a, b)$, instead of $f(x) \propto x^{a - 1} (1 - x)^{b - 1}$.

The fact that $f(x)$ integrates to $1$ implies that

Property. Given $n, k \in \mathbb{N}$, with $k \le n$,

$$\int_0^1 \binom{n}{k} x^k (1 - x)^{n - k} \, dx = \frac{1}{n + 1}.$$

Proof.

Procedure One. Consider throwing independently and uniformly at random $n$ red balls and $1$ blue ball onto the interval $[0, 1]$. Denote $Z$ the number of red balls to the left of the blue ball. The integration gives the probability $\Pr[ Z = k ]$.

Procedure Two. On the other hand, if we 1) throw $n + 1$ red balls uniformly at random onto the interval $[0, 1]$; 2) pick one red ball at random; 3) and replace it by a blue one, then the number of red balls to the left of the blue one follows the same distribution as in procedure one. Hence

$\square$

Corollary. Given $a, b \in \mathbb{N}^+$,

$$\begin{aligned} \beta(a, b) = \int_0^1 x^{a - 1} (1 - x)^{b - 1} \, dx = \frac{(a - 1)! (b - 1)!}{(a + b - 1)!} = \frac{\Gamma(a) \Gamma(b)}{ \Gamma(a + b) }. \end{aligned}$$

Remark: the result can be generalized to $a, b > 0$.

Property.

$$\mathbb{E}[X] = \frac{a}{a + b}.$$

Proof.

$\square$

Property.

$$\mathbb{Var}[X] = \frac{ ab }{(a + b + 1)(a + b)^2}.$$

Proof.

Hence,

$\square$

Property (Beta Binomial Conjugacy). Let $n \in \mathbb{N}^+$. Suppose that $X \sim \mathcal{B}(a, b)$, and $Z \sim \textnormal{Bin}(n, X)$, then for each $k \in \mathbb{N}^+$, $k \le n$, then the posterior probability distribution of $X$ equals $\mathcal{B}(a + k, b + n - k)$.

Proof. Let $f(x \mid Z = k)$ be the posterior density function. Then for each $x \in [0, 1]$,

The last ''$\propto$'' holds since $\Pr[ Z = k ]$ is a constant.

$\square$

Reference

[1] J. Blitzstein and J. Hwang, Introduction to Probability.