Introduction

So far, we’ve dealt primarily with discrete-time systems with discrete alphabets. Now we look to change discrete to continuous.

Let $\mathcal{X}\subset\mathbb{R}$ be a real-valued RV with cdf $F_{X}(x)=P(X\le x), \ x\in\mathbb{R}$ and pdf, $f_{X}$ where $$F_{X}(x)=\int_{-\infty}^{x}f_{X}(t)dt, \ x\in\mathbb{R}$$The support of $X$ with pdf $f_{X}$ is $$S_{X}=\{x\in\mathbb{R}:f_{X}(x)>0\}$$ ## Some intuition Recall that for discrete (finite) alphabet random variables, its entropy has an operational meaning, i.e., it describes the minimal number of codebits per source symbol needed to losslessly describe it. But with a real-valued RV taking values in a continuum, the number of bits would be infinite since we have infinite source symbols.

Intuition

So the operational meaning of the differential entropy is that since $H([X]_{n})$ is the minimum average number of bits needed to losslessly describe $[X]_{n}$, the uniformly quantized X. We then obtain that $h(X)+n$ bits is approximately needed to describe $X$ when uniformly quantizing it with $n$-bit accuracy or $$H([X]_{n})\approx n+h(X)$$for $n$ sufficiently large. So the larger $h(X)$ is, the larger the average number of bits required to describe a uniformly quantized $X$ with $n$-bit accuracy. If $(X,Y)\sim f_{XY}$ on $S_{XY}\subset\mathbb{R}^{2}$ then $$H([X]_{n},[Y]_{m})\approx m+n+h(X,Y)$$for $m,n$ sufficiently large.