FIND ME ON

GitHub

LinkedIn

Convexity or Concavity of Information Measure

🌱

Theorem
InfoTheory

1. The divergence, D(pq)D(p\|q) is convex in the pair (p,q)(p,q) i.e. if (p1,q1(p_1,q_1 and (p2,q2)(p_2,q_2) are two pairs of pmfs defined on X\mathcal{X} then D(λp1+(1λ)p2λq1+(1λ)q2)λD(p1q1)+(1λ)D(p2q2)D(\lambda p_1+(1-\lambda)p_2\|\lambda q_1+(1-\lambda)q_2)\le\lambda D(p_1\|q_1)+(1-\lambda)D(p_2\|q_2) λ[0,1]\forall\lambda\in[0,1]. 2. If XpXX\sim p_X, then H(X)=H(pX)H(X)=H(p_X) is concave in 3. If (X,Y)pXpYX(X,Y)\sim p_Xp_{Y|X}, then I(X;Y)=I(pX,pYX)I(X;Y)=I(p_X,p_{Y|X}) is concave in pXp_X for fixed pYXp_{Y|X} and convex in pYXp_{Y|X} for fixed pXp_X

Linked from