ℹ

Renyi Divergence

Definition (Renyi Divergence)

Given two pmfs pp and qq with common support X\mathscr{X}, and parameter α>0, α≠1\alpha>0, \ \alpha\not=1, the Renyi divergence of parameter α\alpha between pp and qq is: Dα:=1α−1log⁡2(∑a∈Xpα(a)q1−α(a))D_\alpha:=\frac{1}{\alpha-1}\log_2\left(\sum_{a\in\mathscr{X}}p^\alpha(a)q^{1-\alpha}(a)\right)

Lemma (Renyi’s Divergence reduces to Divergence)

lim⁡α→1Dα(p∥q)=D(p∥q)\lim_{\alpha\to1}D_\alpha(p\|q)=D(p\|q)