Penalty Lemma

Lemma

If C\mathcal{C} is the Shannon-Fano Code for the distribution q(xn)q(x^{n}) and XnX^{n} has pmf p(xn)p(x^{n}) then we have the following bound for the redundancy 1nD(pq)R(Cn,p)<1nD(pq)+1n\begin{align*} \\ \frac{1}{n} D(p\|q)\le R(\mathcal{C}_{n},p)< \frac{1}{n}D(p\|q)+ \frac{1}{n} \end{align*} the 1n\frac{1}{n} becomes a 2n\frac{2}{n} if arithmetic coding is used instead.

Cor (Zero Penalty iff Universal Code)

A sequence of Shannon-Fano (or arithmetic) codes {Cn}\{ \mathcal{C}_{n} \} obtained from a sequence of coding distributions {qn}\{ q_{n} \} is universal for a source class P\mathcal{P} if and only if limn1nD(pqn)=0,  pP\lim_{ n \to \infty } \frac{1}{n}D(p\|q_{n})=0, \ \ \forall p\in\mathcal{P}