Let $X$ be a discrete Random Variable with finite Alphabet $\mathscr{X}$ and pmf given by $$p_X(a):=P(X=a), a\in\mathscr{X}$$ The entropy of a discrete Random Variable $X$ with pmf $p_X$ and alphabet $\mathscr{X}$ is denoted by $H(X)$ and given by: $$H(X)=-\sum_{a\in\mathscr{X}}p_X(a)\log_2(p_X(a))$$ ## Intuition Taking the notion of self-information, we can look at entropy as the average information conveyed by some random variable. We can look at entropy as giving us our expected information from a certain distribution rv Examining the definition of entropy $H(X)$ we have that $$\begin{align*} H(X)&=-\sum_{a\in\mathscr{X}}p_X(a)\log_2(p_X(a))\\ &=\sum_{a\in\mathscr{X}}p_X(a)\ I(p_X(a))\\ &=E_{p_X}[I(p_X(x))]\\ &=E_{p_X}[-log_2(p_X(x))] \end{align*}$$ or that $H(X)$ is the expected (statistical average) amount of information one gains about the occurrence of one its $|\mathscr{X}|$ outcomes.

Given $(X,Y)\sim p_{XY}$, the conditional entropy of $X$ given $Y$ is denoted by $H(X|Y)$ and given by $$\begin{align*}H(X|Y)&=E_{p_{XY}}[-\log_2(p_{X|Y}(x|y))]\\&=-\sum_{a\in\mathscr{X}}\sum_{b\in\mathscr{Y}}p_{XY}(a,b) \ \log_2(p_{X|Y}(a|b))\end{align*}$$