FIND ME ON

GitHub

LinkedIn

Penalty Lemma

🌱

Theorem
InfoTheory

Lemma

If C\mathcal{C} is the Shannon-Fano Code for the distribution q(xn)q(x^{n}) and XnX^{n} has pmf p(xn)p(x^{n}) then we have the following bound for the redundancy 1nD(p∄q)≤R(Cn,p)<1nD(p∄q)+1n\begin{align*} \\ \frac{1}{n} D(p\|q)\le R(\mathcal{C}_{n},p)< \frac{1}{n}D(p\|q)+ \frac{1}{n} \end{align*} the 1n\frac{1}{n} becomes a 2n\frac{2}{n} if arithmetic coding is used instead.