Definition (Dynamic Programming)

Consider the problem of minimizing $$J(t_{0},x_{0},u)=J(u)=\int\limits _{t_{0}}^{t_{1}}L(t,x(t),u(t)) \, dt +Q(x(t_{1}))$$where $u:\mathbb{R}\to \mathcal{U},\,(\mathcal{U}\subseteq \mathbb{R}^{m})$, $L:\mathbb{R}\times \mathbb{R}^{n}\times \mathbb{R}^{m}\to \mathbb{R}$ is the running cost (o/w referred to as the Lagrangian), $Q:\mathbb{R}^{n}\to \mathbb{R}$ is the terminal cost, all subject to the following dynamics $$\begin{align*} \dot{x}(t)&= f(x(t),u(t),t)\\ x(t_{0})&= x_{0} \end{align*}$$$\forall t\ge t_{0}$ where $f:\mathbb{R}^{n}\times \mathbb{R}^{m}\times \mathbb{R}\to \mathbb{R}^{n}$. The goal of dynamic programming is to consider a family of minimization problems $$J(t,\mathbf{X},u)=\int\limits _{t}^{t_{1}}L(\tau,x(\tau),u(\tau)) \, d\tau+Q(x(t_{1})) \quad\forall t\in[t_{0},t_{1})$$where $\mathbf{X}\in\mathbb{R}^{n}$ and $x(t)=\mathbf{X}$. Our goal is to derive a dynamic relationship among these problems and solve all of them. To do this we introduce the Value Function $$V(t,\mathbf{X})=\inf_{u[t,t_{1}]}\{ J(t,\mathbf{X},u) \}$$where the control is restricted to future time. We also wish to have $$V(t_{1},\mathbf{X})=Q(\mathbf{X})\quad\forall \mathbf{X}\in\mathbb{R}^{n}$$