The Hamilton-Jacobi-Bellman equation gives a closed form of the Value Function’s partial derivative w.r.t. time: $$-\frac{ \partial V }{ \partial t } (t,\mathbf{X})=\inf_{\mathbf{u}\in\mathcal{U}}\left\{ L(t,\mathbf{X},\mathbf{u})+\frac{ \partial V }{ \partial x } (t,\mathbf{X})\cdot f(\mathbf{X},\mathbf{u},t) \right\}$$subject to $$V(t_{1},\mathbf{X})=Q(\mathbf{X})$$ ## Derivation Consider the dynamics $$\dot{x}(t)=f(t,x(t),u(t))$$then let us take the first-order Taylor Expansion around $\mathbf{X}$ to get $$\begin{align*} x(t+\Delta t)&= x(t)+f(x(t),u(t),t)\Delta t+o(\Delta t)\\ &= \mathbf{X}+f(\mathbf{X},u(t),t)\Delta t+o(\Delta t) \end{align*}$$where $x(t)=\mathbf{X}$. Assuming the Value Function is $C^{1}$ we then take the Taylor expansion of $V$: $$\begin{align*} V(t+\Delta t,x(t+\Delta t))&= V(t,\mathbf{X})+\frac{ \partial V }{ \partial t } (t,\mathbf{X})\Delta t+\frac{ \partial V }{ \partial x } (t,\mathbf{X})\cdot f(\mathbf{X},u(t),t)\Delta t+o(\Delta t) \end{align*}$$where $$\frac{ \partial V }{ \partial x } (t,\mathbf{X})=\nabla V=\begin{bmatrix}\frac{ \partial V }{ \partial x_{1} } \\ \vdots\\\frac{ \partial V }{ \partial x_{n} } \end{bmatrix}$$Finally, we get by applying the Taylor expansion again: $$\int\limits _{t}^{t+\Delta t}L(\tau,x(\tau),u(\tau)) \, d\tau =L(t,\mathbf{X},u(t))\Delta t+o(\Delta t)$$By substitution we can sub these all in to our result from Value Function: $$\begin{align*} V(t,\mathbf{X})&= \inf_{u_{[t,t+\Delta t]}}\left\{ \int\limits _{t}^{t+\Delta t}L(\tau,x(\tau),u(\tau)) \, d\tau +V(t+\Delta t,x(t+\Delta t)) \right\}\\ &= \inf_{u_{[t,t+\Delta t]}}\left\{ L(t,\mathbf{X},u(t))\Delta t+V(t,\mathbf{X})+\frac{ \partial V }{ \partial t }(t,\mathbf{X})\Delta t+\frac{ \partial V }{ \partial \mathbf{X} } (t,\mathbf{X})\cdot f(\mathbf{X},u(t),t)\Delta t+o(\Delta t) \right\} \end{align*}$$As a result we have $$0=\lim_{ \Delta t \to 0 } \frac{1}{\Delta t}\inf_{u_{[t,t+\Delta t]}}\left\{ L(t,\mathbf{X},u(t))\Delta t+\frac{ \partial V }{ \partial t } (t,\mathbf{X})\Delta t+\frac{ \partial V }{ \partial \mathbf{X} } (t,\mathbf{X})\cdot f(\mathbf{X},u(t),t)\cdot\Delta t+o(\Delta t) \right\}$$Note that first $\frac{ \partial V }{ \partial t }(t,\mathbf{X})$ does not depend on the infimum so we can pull it out. Next notice that by taking the limit the control is evaluated solely at time $t$ so we get: $$-\frac{ \partial V }{ \partial t } (t,\mathbf{X})=\inf_{\mathbf{u}\in\mathcal{U}}\left\{ L(t,\mathbf{X},\mathbf{u})+\frac{ \partial V }{ \partial \mathbf{X} } (t,\mathbf{X})\cdot f(\mathbf{X},\mathbf{u},t) \right\}$$