Orthogonality Principle

Recall

In our closed-loop linear predictor we defined the error as en=Xni=1maiXnie_{n}=X_{n}-\sum_{i=1}^{m}a_{i}X_{n-i}We then have that E[enXnj]=E[(Xni=1maiXni)Xnj]=E[XnXnj]k=1maiE[XnkXnj]\begin{align*} E[e_{n}X_{n-j}]&=E\left[ \left( X_{n}-\sum_{i=1}^{m}a_{i}X_{n-i} \right)X_{n-j} \right]\\ &=E[X_{n}X_{n-j}]-\sum_{k=1}^{m}a_{i}E[X_{n-k}X_{n-j}] \end{align*}We see that from (*), a1,,ama_{1},\dots,a_{m} is optimal if and only if E[enXnj]=0E[e_{n}X_{n-j}]=0, j=1,,mj=1,\dots,m.

Theorem (Orthogonality Principle)

The linear predictor X^n=k=1makXnk\hat{X}_{n}=\sum_{k=1}^{m}a_{k}X_{n-k} is optimal in the MSE sense if and only if the prediction error is orthogonal to all XnjX_{n-j} i.e.X^n optimal     (XnX^n)Xnj   j=1,,m\hat{X}_{n}\text{ optimal }\iff(X_{n}-\hat{X}_{n})\perp X_{n-j} \ \ \ j=1,\dots,m