FIND ME ON

GitHub

LinkedIn

Shannon-Fano-Elias Code

🌱

InfoTheory

Construction

In SFE coding, we order our alphabet X={a1,,am}\mathcal{X}=\{ a_{1},\dots,a_{m} \} such that a1<<ama_{1}<\dots<a_{m}.

We induce a lexicographical ordering (i.e. think alphabetical order like in a dictionary) on Xn\mathcal{X}^n for n2n\ge2: yn=(y1,..,yn)<(x1,,xn)=xn    y1<x1y^{n}=(y_{1},..,y_{n})<(x_{1},\dots,x_{n})=x^{n}\iff y_{1}<x_{1}or k{2,,n}\exists k\in\{ 2,\dots,n \} s.t. yi=xi,  i=1,..,k1 and yk<xky_{i}=x_{i}, \ \ i=1,..,k-1\text{ and }y_{k}<x_{k} and so on.

Let p(xn)p(x^{n}) be a pmf on Xn\mathcal{X}^{n} such that F^(xn)=yn<xnp(yn),  F(xn)=ynxnp(yn)\hat{F}(x^{n})=\sum_{y^{n}<x^{n}}p(y^{n}), \ \ {F}(x^{n})=\sum_{y^{n}\le x^{n}}p(y^{n})The tag of xnx^{n} is defined as Tˉ(xn)=F^(xn)+12p(xn)\bar{T}(x^{n})=\hat{F}(x^{n})+\frac{1}{2}p(x^{n})i.e. the midpoint of the interval [F^(xn),F(xn))[\hat{F}(x^{n}),F(x^{n})).

The tag of a source word xnXnx^{n}\in\mathcal{X}^{n} is defined as Tˉ(xn)=F^(xn)+12p(xn)\bar{T}(x^{n})=\hat{F}(x^{n})+\frac{1}{2}p(x^{n}) where F^(xn)=yn<xnp(yn)\hat{F}(x^{n})=\sum_{y^n<x^n}p(y^{n})

The Shannon-Fano-Elias Code C\mathcal{C} on Xn\mathcal{X}^{n} is defined by C(xn)=bl(xn)(Tˉ(xn))\mathcal{C}(x^{n})=b^{l(x^{n})}(\bar{T}(x^{n}))where l(xn)=logp(xn)+1l(x^{n})=\lceil -\log p(x^{n}) \rceil +1That is, the codeword associated with xnx^n is the binary representation of the tag of xnx^n, Tˉ(xn)\bar{T}(x^{n}) truncated to logp(xn)+1\lceil -\log p(x^n) \rceil+1 bits.

Intuition

So by the way we define the tag we’re defining it in a way that for each xnx^{n} in the set of messages, their tag will be inside of a disjoint interval, [F^(xn),F(xn))[\hat{F}(x^{n}),F(x^{n})) from all others xnx^{n}s.

The is a prefix code with average code length lˉ(Cxn)=E[l(Cxn)]<H(Xn)+2\bar{\mathscr{l}}(\mathcal{C}_{x^n})=E[{\mathscr{l}(\mathcal{C}_{x^{n}})}]<H(X^{n})+2or average code rate Rˉn(C)1nH(Xn)+2n\bar{R}_{n}(\mathcal{C})\le \frac{1}{n}H(X^{n})+ \frac{2}{n} and if (Xn)nN(X_{n})_{n\in\mathbb{N}} is stationary then limnRˉn(C)=Hˉ(X)\lim_{ n \to \infty } \bar{R}_{n}(\mathcal{C})=\bar{H}(X)