Cross-Entropy

Definition (Cross entropy)

Given two pmfs $p$ and $q$ on the same alphabet $\mathscr{X}$ , the Cross-Entropy between $p$ and $q$ , denoted as $H(p;q)$ is $H(p;q):=\sum_{a\in\mathscr{X}}p(a)\log_2\frac{1}{q(a)}$

Remark

Cross-Entropy is a quantity intimately connected to divergence: $\begin{align*} D(p\|q)&=\sum_{a\in\mathscr{X}}p(a)\log_2\left(\frac{p(a)}{q(a)}\right)\\ &=\sum_{a\in\mathscr{X}}p(a)\log_2p(a)+\sum_{a\in\mathscr{X}}p(a)\log_2\frac{1}{q(a)}\\ &=-H(X)+H(p;q) \end{align*}$

Intuition

Where divergence essentially tells us the difference in entropy between using $p$ to encode $p$ and using $q$ to encode $p$ . Cross-Entropy tells us the total number of expected bits (or entropy) needed to when symbols are generated from $p$ but $q$ is used to encode them.

Proposition (Entropy vs Cross-Entropy)

Since $D(p\|q)\ge0$ , then by the above remark we have $0\le D(p\|q)=-H(p)+H(p;q)$ which implies $\begin{align*} D(p\|q)&\le H(p;q)\\ H(p)&\le H(p;q) \end{align*}$

Linked from

Differential Cross-Entropy

Cross-Entropy