ℹ

Renyi Entropy

Definition (Renyi Entropy)

Given parameter α>0, α≠1\alpha>0, \ \alpha\not=1, a RV X∼pXX\sim p_X, then the Renyi entropy with parameter α\alpha is Hα(X)=11−αlog⁡2(∑a∈XpX(a)α)=α1−αlog⁡2(∥pX∥α)\begin{align*} H_\alpha(X)&=\frac{1}{1-\alpha}\log_2\left(\sum_{a\in\mathscr{X}}p_X(a)^\alpha\right)\\ &=\frac{\alpha}{1-\alpha}\log_2\left(\|p_X\|_\alpha\right) \end{align*} where ∥pX∥α\|p_X\|_\alpha is the “α\alpha-norm of pXp_X” or ∥pX∥α=[∑a∈XpX(a)α]1α\|p_X\|_\alpha=\left[\sum_{a\in\mathscr{X}}p_X(a)^\alpha\right]^{\frac{1}{\alpha}}

Lemma (Renyi Entropy reduces to Entropy)

lim⁡α→1Hα(X)=H(X)\lim_{\alpha\to1}H_\alpha(X)=H(X)

Linked from