Self-Information

NAVIGATION

Home

Research

Bookshelf

Garden

FIND ME ON

GitHub

Home

Research

Bookshelf

Garden

ℹ

Self-Information

Definition (Self-information)

Let $E$ denote an event with probability $p>0$ of occurring. We call $I(E)$ or $I(p)$ as it is a function of p, the self-information of $E$ and use it to represent the “amount of information” one gains about event $E$ when learning that $E$ has occurred.

Remark

Equivalently, we can also think of it as a measure of the “amount of uncertainty” one had prior to the occurrence of event $E$ .

Proposition (Properties of $I(p)$ )

Certain events are not surprising: If some event $E$ will most definitely happen, $p(E)=1$ , then that event occurring should provide us with no surprise (or new information): $H(E)=0$ .
Impossible events are infinitely surprising: If some event $E$ , has zero chance of occurring, $p(E)=0$ , then we should be infinitely surprised that the event is occurring $H(E)=\infty$ .
Non-Increasing: $I(p)$ should be non-increasing in p (i.e. the less likely event $E$ is, the more information one gains from it happening).
Continuity: $I(p)$ should be continuous in p. Intuitively, one would expect that a small change in p corresponds to a small change in the amount of information about $E$ .
Continuity of Independence: If $E_1$ and $E_2$ are independent with probabilities $p_1>0$ and $p_2>0$ , respectively, then $I(E_1\cap E_2)=I(p_1*p_2)=I(p_1)+I(p_2)$ This property is “reasonable” as $E_1$ and $E_2$ are independent.

Remark

$I(1)=0$ $I (1) = 0$
- Setting $p_1=p_2$ yields $I(1*1)=I(1)+I(1)$ which implies that $I(1)=0$
$I(p)\ge 0$ $I (p) \geq 0$ $\forall p$ $\forall p$
- By the first remark and the fact that $I(p)$ is non-increasing we see that this holds true.

Theorem (Representation of $I(p)$ )

The only function $I(p)$ , $0\le p\le1$ , satisfying properties 1-5 above is given by $I(p)=-c\log_b(p)$ where $c>0$ and $b>1$ are constants (b is for base unit).

Remark

We usually set $c=1$ and $b=\{2,e\}$ in this course.

Proposition (b unit table)

$\begin{array} {|r|r|}\hline b & \text{units of }I(p) \\ \hline 2 & \text{bits} \\ \hline e & \text{nats} \\ \hline 3 & \text{ternary units} \\ \hline q & \text{q-ary digits} \\ \hline \end{array}$

Linked from

Redundancy

Asymptotic Equipartition Property

Entropy

Mutual Information