In our Closed-loop Predictive Quantization system one may wonder, “how are the predictor coefficients, $a_{i}$, chosen?”

1. We may first attempt to minimize the closed-loop Mean Squared Error Distortion prediction error $$E[e_{n}^{2}]=E\left[ \left( X_{n}-\sum^{m}_{i=1}a_{i}\hat{X}_{n-i} \right)^{2} \right]$$but this is too computationally complex as for large sequences each $\hat{X}_{n-i}$ will depend on previous $a_{i}$ in a highly linear fashion.
2. We may instead choose $a_{i}$ to minimize the open-loop error $$E\left[ \left( X_{n}-\sum^{m}_{i=1}a_{i}X_{n-i} \right)^{2} \right]$$so now we state the problem formally…

Problem

Given RVs $X_{n-m},X_{n-m+1},\dots,X_{n-1},X_{n}$ (all have finite variance), determine their predictor coefficients, $a_{i}$ by minimizing our prediction error $$\min E\left[ \left( X_{n}-\sum^{m}_{i=1}a_{i}X_{n-i} \right)^{2} \right]$$

Solution

Define $$g(a_{1},\dots,a_{m})=E\left[ \left( X_{n}-\sum^{m}_{i=1}a_{i}X_{n-i} \right)^{2} \right]$$We must find $$\underset{ a_{1},\dots,a_{m} }{ \arg\min } \ g(a_{1},\dots,a_{m})$$Since $$\begin{align*} &E\left[ \left( X_{n}-\sum^{m}_{i=1}a_{i}X_{n-i} \right)^{2} \right]\\ &=E[X_{n}^{2}]+\sum_{i=1}^{m}\sum_{k=1}^{m}a_{i}a_{k}E[X_{n-i}X_{n-k}]-2\sum_{i=1}^{m}a_{i}E[X_{n}X_{n-i}] \end{align*}$$now we derive with respect to $a_{j}$ $$\frac{ \partial g }{ \partial a_{j} } =2\sum_{k=1}^{m}a_{k}E[X_{n-j}X_{n-k}]-2E[X_{n}X_{n-j}]$$ Setting $\frac{ \partial g }{ \partial a_{j} }=0$ $\forall j$ gives $m$ linear equations (note this works because distortion is convex).

System of equations

$$\tag{*}\sum_{k=1}^{m}a_{k}E[X_{n-j}X_{n-k}]=E[X_{n}X_{n-j}], \ \ \ j=1,\dots,m$$Let $r_{ik}=E[X_{n-i}X_{n-k}]$ and $$\mathbf{R}=\begin{bmatrix}r_{11} & r_{12} & \dots & r_{1m} \\ r_{21} & r_{22} & \dots & r_{2m} \\ \vdots & \vdots & \ddots & \vdots \\ r_{m1} & r_{m2} & \dots & r_{mm} \end{bmatrix}\quad \mathbf{v}=\begin{bmatrix}r_{01} \\ r_{02} \\ \vdots \\ r_{0m}\end{bmatrix}$$From $(*)$ the optimal $\mathbf{a}=(a_{1},\dots,a_{m})^{T}$ must solve $$\mathbf{Ra}=\mathbf{v}$$Solution always exists but is only unique if $\mathbf{R}$ is Invertible $$\mathbf{a}=\mathbf{R}^{-1}\mathbf{v}$$

Remark

The autocorrelation matrix $\mathbf{R}$ is symmetric (i.e. $r_{ij}=r_{ji}$) It is also Positive Semidefinite (or definite) i.e. $$\mathbf{x}^{T}\mathbf{R}\mathbf{x}\ge 0 \ \ \ \forall \mathbf{x}=(x_{1},\dots,x_{m})^{T}$$