Differential Cross-Entropy

Definition (Differential cross-entropy)

XfX,YfYX\sim f_{X},Y\sim f_{Y} with SXSYRS_{X}\subset S_{Y}\subset \mathbb{R}, the differential cross-entropy between fXf_{X} and fYf_{Y} is h(fX;fY)=SXfX(t)log21fY(t)dth(f_{X};f_{Y})=\int_{S_{X}}f_{X}(t)\log_{2} \frac{1}{f_{Y}(t)}dt

Remark

D(fXfY)=h(X)+h(fX;fY)0D(f_{X}\|f_{Y})=-h(X)+h(f_{X};f_{Y})\ge0\therefore h(fX;fY)h(X)h(f_{X};f_{Y})\ge h(X)