Theorem

For the LQ problem, let $J_{N}=Q_{N}$ and $$P_{t}=Q+A^{T}P_{t+1}A-A^{T}P_{t+1}+B(R+B^{T}P_{t+1}B)^{-1}B^{T}P_{t+1}A$$with final condition $P_{N}=Q_{N}$.

The optimal cost is $$\inf_{\gamma\in\Gamma_{A}}J(x,\gamma)=J(x_{0})=x_{0}^{T}P_{0}x_{0}+E[w_{t}^{T}P_{t+1}w_{t}]$$with our optimal control being $$\gamma_{t}^{*}=-(B^{T}P_{t+1}B+R)^{-1}B^{T}P_{t+1}Ax_{t}$$