Information Capacity

Definition (Information Capacity)

Given a DMC $(\mathcal{X},\mathcal{Y},Q=[P_{XY}])$ , its Information Capacity, $C$ , is defined as: $C=\max_{p_{X}}I(X;Y)$ where maximization is over all possible input distributions $p_{X}$ .

Remark

$C$ is well-defined since $I(X;Y)=I(P_{X},P_{Y|X})$ is a concave and continuous function of $p_{X}$ (by concavity of information measure) and the set of $p_{X}$ is compact (closed and bounded) in $\mathbb{R}^{|\mathcal{X}|}$ ; thus $I(X;Y)$ admits a maximum. Also since $0\le I(X;Y)\le \min\{\log|\mathcal{X}|,\log|\mathcal{Y}|\},$ then $0\le C\le \min\{\log|\mathcal{X}|,\log|\mathcal{Y}|\}$
Will see via Shannon’s Channel Coding Theorem for the DMC that $C$ is the DMC’s “operational capacity”

Problem

Given a DMC $(\mathcal{X},\mathcal{Y},Q=[P_{XY}])$ , determine the Information Capacity $C=\max_{p_{X}}I(X;Y)$ First note that necessary and sufficient conditions for finding $C$ can be shown since $I(X;Y)=I(p_{X},p_{Y|X})$ is concave in $p_{X}$ ; but they might be hard to use as they require “guessing” a priori the optimal input distribution.

$\therefore$ In general, $C$ does not admit a closed-form expression (in terms of channel parameters), unless if the channel exhibits certain “symmetry” properties.

Linked from

Binary Erasure Channel

Binary Symmetric Channel

Binary Symmetric Erasure Channel

Information Capacity

Lossless Joint Source-Channel Coding Theorem

Shannon's Channel Coding Theorem for the DMC

Symmetric Channel

Lossy Source-Channel Coding Theorem

Shannon Limit

Capacity of Parallel Gaussian Channels

Information Capacity with Input Cost

Upper Bound on Channel Capacity