Theorem

Consider the finite horizon optimization problem $$J(X,\gamma)=E_{x}^{\gamma}\left[ \sum_{k=0}^{N-1}c(X_{k},U_{k})+c_{N}(X_{N}) \right]$$If $\exists J_{0},\dots,J_{N-1},f_{0},\dots,f_{N-1}$ where $$J_{N}(x)=c_{N}(x)$$and for $0\le t\le N-1$ $$\begin{align*} J_{t}(x)&=\min_{u\in\mathbb{U}}(c(x,u)+E[J_{t+1}(x_{t+1})|x_{t}=x,u_{t}=u])\\ &=c(x,f_{t}(x))+E[J_{t+1}(x_{t+1})|x_{t}=x,u_{t}=f_{t}(x)] \end{align*}$$then we have that $$\inf_{\gamma\in\Gamma_{A}}J_{N}(x)=J_{0}(x)$$and $\gamma^{*}=\{ f_{0},\dots,f_{N-1} \}$ is optimal.