Lemma (Dirichlet Mixture Distribution)
The coding distribution corresponding to the Dirichlet (α1,…,αm) pdf is given by q(xn)=Θ∫pθ(xn)fα1,…,αm(θ)dθ=∏j=1n(n+∑i=1mαi−j)∏i=1m∏j=1n(i∣xn)(n(i∣xn)+αi−j)where ∏j=1n(i∣xn)(n(i∣xn)+αi−j)=1 if n(i∣xn)=0.
We consider two special cases, one where αi=1 and the other when αi=21.
Lemma (Krichevsky and Trofimov Coding Distribution)
For all 1≤t≤n define q(i∣xt−1)=t−1+2mn(i∣xt−1)+21Then
- q(i∣xt−1) is a conditional pmf on X given xt−1
- t=1∏nq(xt∣xt−1)=∏j=1n(n+2m−j)∏i=1m∏j=1n(i∣xn)(n(i∣xn)+21−j)=q(xn)
Theorem
Let qn be the on Xn, where ∣X∣=m. Then for any pmf p(x) on X and iid source distribution pn(xn)=∏i=1np(xi) n1D(pn∥qn)≤2nm−1logn+nK0, ∀n≥1for some constant K0.
Cor (Max Redundancy on KT)
Let P denote the class of iid sources on X. If Cn is the arithmetic code for on Xn, then its redundancy satisfies: p∈PmaxR(Cn,p)≤2nm−1logn+O(n1)This implies that KT coding distribution is minimax optimal.