Definition

Pasted image 20240321123912.png - Let $\mathbf{A}$ be a $k\times k$ orthogonal matrix. - $\mathbf{X}=(X_{1},\dots,X_{k})^{T}$ be a random vector representing our input and - $\mathbf{AX}=\mathbf{Y}=(Y_{1},\dots,Y_{k})^{T}$ represent the transform coefficients. - For $i=1,\dots,k$ we define $Q_{i}$ as a $N_{i}$-level scalar quantizer. - Finally, the quantized coefficients are defined as $\mathbf{\hat{Y}}=(Q_{1}(Y_{1}),\dots,Q_{k}(Y_{k}))^{T}$ and; - Our quantized output is $\mathbf{A}^{-1}\mathbf{\hat{Y}}=\mathbf{\hat{X}}=(\hat{X}_{1},\dots,\hat{X}_{k})^{T}$

The end-to-end MSE distortion in transform coding is defined as $$D_{\text{tc}}=\sum_{i=1}^{k}E[(X_{i}-\hat{X}_{i})^{2}]=E[\lVert \mathbf{X}-\mathbf{\hat{X}} \rVert ^{2}]$$