Jensen-Shannon Divergence

Definition (Jensen-Shannon Divergence)

Given two pmfs $p$ and $q$ on the same alphabet $\mathscr{X}$ , the Jensen-Shannon Divergence is defined as $D_{JS}(p\|q):=\frac{1}{2}D(p\|\mathcal{M})+\frac{1}{2}D(q\|\mathcal{M})$ where $\mathcal{M}(a):=\frac{p(a)+q(a)}{2}$

Intuition

This is essentially a symmetric version of divergence hence it’s more widely applicable and does not place one distribution as the focal point or reference point.

Remark

The Jensen-Shannon Divergence satisfies the nonnegative and symmetric properties of a distance but not the triangular inequality (it’s square root does).