Q.9.17 - A First Course in Probability

Question

Show that for any discrete random variable $X$ and function$f$

 $H\left(f\right(X\left)\right)\le H\left(X\right)$

Step-by-Step Solution

Answer

The given statement is proved below.

Step 1: Given Information

We have to prove that

$H\left(f\right(X\left)\right)\le H\left(X\right)$

Step 2: simplify

Consider two random variables, $X$and$f\left(X\right)$ . Using the formula about the entropy of two random variables, we have that

$H\left(X,f\left(X\right)\right)=H\left(X\right)+H_X\left(f\left(X\right)\right)$

Observe that knowing  $X$, random variable $f\left(X\right)$ becomes absolutely deterministic, i.e. there is no more uncertainty about its value. Hence $H_X\left(f\left(X\right)\right)=0$ which implies that $H\left(X,f\left(X\right)\right)=H\left(X\right)$. On the other hand, we can also write

$H\left(X,f\left(X\right)\right)=H\left(f\left(X\right)\right)+H_{f\left(X\right)}\left(X\right)\ge H\left(f\left(X\right)\right)$

which finally implies that

$H\left(f\left(X\right)\right)\le H\left(X\right)$