Q. 9.15 - A First Course in Probability

Question

A coin having probability p = 2 3 of coming up heads is flipped 6 times. Compute the entropy of the outcome of this experiment.

Step-by-Step Solution

Answer

The entropy of the outcome of this experiment is$H\left(\mathbb{X}\right)=5.5098$.

Step 1: Given Information

We have given the probability of the coin $p=\frac{2}{3}$.

Step 2: Simplify

Considering random vector $\mathbb{X}=\left(X_1,\dots ,X_6\right)$ where $X_i$ are independent equally distributed variables with distribution which given as

$X_i~\text{Bern}\left(\frac{2}{3}\right)$

Calculating the entropy of $\mathbb{X}$.

$H\left(\mathbb{X}\right)=-\sum _{x\in {\left(0,1\right)}^6}p\left(x\right)\text{log}p\left(x\right)$

This sum contains $2^6$summands. Then,

$$\begin{gathered} \sum _{x\in {\left(0,1\right)}^6}=p\left(x\right)\text{log}p\left(x\right)=-\left(\begin{array}{c}6\\ 0\end{array}\right){\left(\frac{1}{3}\right)}^6\text{log}{\left(\frac{1}{3}\right)}^6 \\ =-\left(\begin{array}{c}6\\ 1\end{array}\right){\left(\frac{1}{3}\right)}^5\frac{2}{3}\text{log}{\left(\frac{1}{3}\right)}^5\frac{1}{6} \\ = -\left(\begin{array}{c}6\\ 6\end{array}\right){\left(\frac{2}{3}\right)}^6\text{log}{\left(\frac{2}{3}\right)}^6 \end{gathered}$$

Hence,

$H\left(\mathbb{X}\right)=5.5098$