Definition (Karhunen-Loeve Transform)
Let R X \mathbf{R_{X}} R X be the autocorrelation matrix for our input X \mathbf{X} X . We first order the eigenvalues of R X \mathbf{R_{X}} R X s.t. λ 1 ≥ ⋯ ≥ λ k ≥ 0 \lambda_{1}\ge \dots\ge \lambda_{k}\ge 0 λ 1 ≥ ⋯ ≥ λ k ≥ 0 and let u 1 , … , u k \mathbf{u}_{1},\dots,\mathbf{u}_{k} u 1 , … , u k be the corresponding orthogonal eigenvectors . We then normalize u i \mathbf{u}_{i} u i to unit length ∥ u i ∥ = 1 , i = 1 , … , k \lVert \mathbf{u}_{i} \rVert =1, \quad i=1,\dots,k ∥ u i ∥ = 1 , i = 1 , … , k Now we define U = [ u 1 u 2 … u k ] \mathbf{U}=\begin{bmatrix}\mathbf{u}_{1}&\mathbf{u}_{2}&\dots&\mathbf{u}_{k}\end{bmatrix} U = [ u 1 u 2 … u k ] and let T = U T = [ u 1 T u 2 T ⋮ u k T ] \mathbf{T}=\mathbf{U}^{T}=\begin{bmatrix}\mathbf{u}_{1}^{T}\\\mathbf{u}_{2}^{T}\\\vdots\\\mathbf{u}_{k}^{T}\end{bmatrix} T = U T = u 1 T u 2 T ⋮ u k T We call T T T the Karhunen-Loeve transform (KLT) matrix for X \mathbf{X} X .
Theorem (Finite Variance = Autocorrelation symmetric + positive semidefinite)
Let X = ( X 1 , … , X k ) T \mathbf{X}=(X_{1},\dots,X_{k})^{T} X = ( X 1 , … , X k ) T be a random vector having finite variance . Then the k × k k\times k k × k autocorrelation matrix R X = { E [ X i X j ] } \mathbf{R_{X}}=\{ E[X_{i}X_{j}] \} R X = { E [ X i X j ]} is symmetric and positive semidefinite .
Theorem (Linear transform for autocorrelation matrices)
If X = ( X 1 , … , X k ) T \mathbf{X}=(X_{1},\dots,X_{k})^{T} X = ( X 1 , … , X k ) T , A \mathbf{A} A is a k × k k\times k k × k matrix, and Y = A X \mathbf{Y}=\mathbf{AX} Y = AX , then R Y = A R X A T \mathbf{R_{Y}}=\mathbf{AR_{X}A}^{T} R Y = A R X A T
Theorem (Determinants for linearly transformed autocorrelation matrices)
If Y = A X \mathbf{Y}=\mathbf{AX} Y = AX then det R Y = det A 2 det R X \det \mathbf{R_{Y}}=\det \mathbf{A}^{2}\det \mathbf{R_{X}} det R Y = det A 2 det R X If A \mathbf{A} A is orthogonal then det R Y = det R X \det \mathbf{R_{Y}}=\det \mathbf{R_{X}} det R Y = det R X
Proposition (KL Transform Decorrelates X)
The transformation Y = T X \mathbf{Y}=\mathbf{TX} Y = TX (where T \mathbf{T} T is KLT ) decorrelates X \mathbf{X} X , i.e., Y = ( Y 1 , … , Y k ) T \mathbf{Y}=(Y_{1},\dots,Y_{k})^{T} Y = ( Y 1 , … , Y k ) T satisfies E [ Y i Y j ] = 0 ∀ i ≠ j E[Y_{i}Y_{j}]=0\quad\forall i\not=j E [ Y i Y j ] = 0 ∀ i = j