Karhunen-Loeve Transform

Definition (Karhunen-Loeve Transform)

Let RX\mathbf{R_{X}} be the autocorrelation matrix for our input X\mathbf{X}. We first order the eigenvalues of RX\mathbf{R_{X}} s.t. λ1λk0\lambda_{1}\ge \dots\ge \lambda_{k}\ge 0and let u1,,uk\mathbf{u}_{1},\dots,\mathbf{u}_{k} be the corresponding orthogonal eigenvectors. We then normalize ui\mathbf{u}_{i} to unit length ui=1,i=1,,k\lVert \mathbf{u}_{i} \rVert =1, \quad i=1,\dots,kNow we define U=[u1u2uk]\mathbf{U}=\begin{bmatrix}\mathbf{u}_{1}&\mathbf{u}_{2}&\dots&\mathbf{u}_{k}\end{bmatrix} and let T=UT=[u1Tu2TukT]\mathbf{T}=\mathbf{U}^{T}=\begin{bmatrix}\mathbf{u}_{1}^{T}\\\mathbf{u}_{2}^{T}\\\vdots\\\mathbf{u}_{k}^{T}\end{bmatrix} We call TT the Karhunen-Loeve transform (KLT) matrix for X\mathbf{X}.

Remarks

Theorem (Finite Variance = Autocorrelation symmetric + positive semidefinite)

Let X=(X1,,Xk)T\mathbf{X}=(X_{1},\dots,X_{k})^{T} be a random vector having finite variance. Then the k×kk\times k autocorrelation matrix RX={E[XiXj]}\mathbf{R_{X}}=\{ E[X_{i}X_{j}] \} is symmetric and positive semidefinite.

Theorem (Linear transform for autocorrelation matrices)

If X=(X1,,Xk)T\mathbf{X}=(X_{1},\dots,X_{k})^{T}, A\mathbf{A} is a k×kk\times k matrix, and Y=AX\mathbf{Y}=\mathbf{AX}, then RY=ARXAT\mathbf{R_{Y}}=\mathbf{AR_{X}A}^{T}

Theorem (Determinants for linearly transformed autocorrelation matrices)

If Y=AX\mathbf{Y}=\mathbf{AX} then detRY=detA2detRX\det \mathbf{R_{Y}}=\det \mathbf{A}^{2}\det \mathbf{R_{X}}If A\mathbf{A} is orthogonal then detRY=detRX\det \mathbf{R_{Y}}=\det \mathbf{R_{X}}

Proposition (KL Transform Decorrelates X)

The transformation Y=TX\mathbf{Y}=\mathbf{TX} (where T\mathbf{T} is KLT) decorrelates X\mathbf{X}, i.e., Y=(Y1,,Yk)T\mathbf{Y}=(Y_{1},\dots,Y_{k})^{T} satisfies E[YiYj]=0ijE[Y_{i}Y_{j}]=0\quad\forall i\not=j

Remark

We see that E[Yi2]=λiE[Y_{i}^{2}]=\lambda_{i} of RX\mathbf{R_{X}} Also we see that any k×kk\times k positive semidefinite R\mathbf{R} with eigenvalues λ1,,λk\lambda_{1},\dots,\lambda_{k} can be written as R=UΛUT\mathbf{R}=\mathbf{U\Lambda U}^{T} where U\mathbf{U} is the orthogonal and Λ=diag(λ1,,λk)\mathbf{\Lambda}=\text{diag}(\lambda_{1},\dots,\lambda_{k}) is diagonal.

Linked from