Hilbert Space

Definition (Hilbert space)

An F\mathbb{F}-Hilbert Space is an F\mathbb{F}-inner product space that is a Complete.

Remark

Any Hilbert Space is a Banach Space.

Remark (Why Hilbert spaces?)

  1. They allow us geometric insight on a set of signals
  2. If a Hilbert space is Separable then there exists countably (or finitely) many Orthonormal vectors which can be used as a Basis.
  3. This formulation allows us to develop approximations of signals using a finite number of basis signals
  4. We can state a projection theorem which certifies optimality in a wide variety of applications.

Theorem (2.2.3)

Let HH be a Hilbert space and BB a subspace of HH. Consider the problem of approximating some point xHx\in H using a point from BB: infmBxm\inf_{m\in B}\lVert x-m \rVert

  1. A necessary and sufficient condition for mBm^{*}\in B to be the minimizing element in BB so that infmBxm=xm(1)\inf_{m\in B}\lVert x-m \rVert =\lVert x-m^{*} \rVert \tag{1}or equivalently xmxyyB\lVert x-m^{*} \rVert \le \lVert x-y \rVert \quad \forall y\in Bis that xmx-m^{*} is orthogonal to BB. If mm^{*} exists, then it is also unique.
  2. Let HH be a Hilbert space and BB be a closed subspace of HH. For any vector xHx\in H, there is a unique vector mBm^{*}\in B satisfying (1)(1).

\begin{proof} We first show 1. Let xHx\in H, m0Bm_{0}\in B. Want to show that if xm0x-m_{0} is not Orthogonal vector to BB then it cannot be optimal.

Suppose m0Bm_{0}\in B is such that mB\exists m\in B with <xm0,m>>0\left< x-m_{0},m \right>>0, i.e., xm0x-m_{0} is not orthogonal to BB.

WLOG, take m=1\lVert m \rVert=1 and <xm0,m>=δ\left< x-m_{0},m \right>=\delta.

We can show that with m1=m0+δmm_{1}=m_{0}+\delta m, we will have xm12=xm1,xm12=xm0δm,xm0δm=xm0,xm0δm,xm0δxm0,m+δ2m,m=xm022δ2+δ2=xm02δ2<xm02\begin{align*} \lVert x-m_{1} \rVert ^{2}&= \sqrt{ \langle x-m_{1}, x-m_{1} \rangle }^{2}\\ &= \langle x-m_{0}-\delta m, x-m_{0}-\delta m \rangle \\ &= \langle x-m_{0}, x-m_{0} \rangle -\delta \langle m, x-m_{0}\rangle-\delta \langle x-m_{0}, m \rangle +\delta^{2}\langle m, m \rangle \\ &= \lVert x-m_{0} \rVert ^{2}-2\delta^{2}+\delta^{2}\\ &= \lVert x-m_{0} \rVert ^{2}-\delta^{2}\\ &< \lVert x-m_{0} \rVert ^{2} \end{align*}and thus m0m_{0} cannot be a minimizer of (1)(1).

We now show that if xmx-m^{*} is orthogonal to BB, then it is optimal. If <xm,M>=0\left< x-m^{*},M \right> =0, we have that for any mBm\in B
xm2=xm+mm2=xm2+mm2xm2\lVert x-m \rVert ^{2}=\lVert x-m^{*}+m^{*}-m \rVert ^{2}=\lVert x-m^{*} \rVert ^{2}+\lVert m^{*}-m \rVert ^{2}\ge \lVert x-m^{*} \rVert^{2} since for xyx\perp y we have that x+y2=x2+y2\lVert x+y \rVert^{2}=\lVert x \rVert^{2}+\lVert y \rVert^{2}. This yields a contradiction, proving 1.

We now show 2. Let δ=infmBxm\delta=\inf_{m\in B}\lVert x-m \rVert Let {mk}kN\{ m_{k} \}_{k\in \mathbb{N}} be s.t. xmkδ\lVert x-m_{k} \rVert \to\deltaand observe that xmk+xmn,xmk+xmn+xmk(xmn),xmk(xmn)=xmk,xmk+xmn+xmn,xmk+xmn+xmk,xmk(xmn)+(xmn),xmk(xmn)=2xmk,xmk+2xmn,xmn\begin{align*} &\langle x-m_{k}+x-m_{n}, x-m_{k}+x-m_{n} \rangle +\langle x-m_{k}-(x-m_{n}), x-m_{k}-(x-m_{n}) \rangle \\ &= \langle x-m_{k}, x-m_{k} +x-m_{n}\rangle +\langle x-m_{n}, x-m_{k}+x-m_{n} \rangle \\ &\quad\quad\quad+ \langle x-m_{k}, x-m_{k}-(x-m_{n}) \rangle +\langle -(x-m_{n}), x-m_{k}-(x-m_{n}) \rangle \\ &= 2\langle x-m_{k}, x-m_{k} \rangle +2\langle x-m_{n}, x-m_{n} \rangle \end{align*}Write xmk+xmn,xmk+xmn=2(xmk+mn2),2(xmk+mn2)=4xmk+mn2,xmk+mn2\begin{align*} \langle x-m_{k}+x-m_{n}, x-m_{k}+x-m_{n} \rangle &= \left\langle 2\left( x- \frac{m_{k}+m_{n}}{2} \right), 2\left( x- \frac{m_{k}+m_{n}}{2} \right) \right\rangle \\ &= 4\left\langle x- \frac{m_{k}+m_{n}}{2}, x- \frac{m_{k}+m_{n}}{2} \right\rangle \tag{🚧} \end{align*}Since mk+mn2B\frac{m_{k}+m_{n}}{2}\in B then by our definition of δ\delta we have that: xmk+mn2δ\left\lVert x- \frac{m_{k}+m_{n}}{2} \right\rVert \ge\deltathen we square both sides then multiply by 44 4xmk+mn224δ24\left\lVert x- \frac{m_{k}+m_{n}}{2} \right\rVert ^{2}\ge 4\delta^{2} and since the LHS is exactly (🚧)(🚧) then by applying Parallelogram law we get mkmn22xmn2+2xmk24δ2\lVert m_{k}-m_{n} \rVert ^{2}\le 2\lVert x-m_{n} \rVert ^{2}+2\lVert x-m_{k} \rVert ^{2}-4\delta^{2}As a result, as xmn2δ2\lVert x-m_{n} \rVert^{2}\to\delta^{2}, we have that mkm_{k} is Cauchy. Since BB is closed, it has a limit, call the limit m~\tilde{m}. We claim that the limit is optimal and hence m~=m\tilde{m}=m^{*}. Conside the difference: xmn,xmnxm~,xm~\langle x-m_{n}, x-m_{n} \rangle -\langle x-\tilde{m}, x-\tilde{m} \rangle We claim that the difference goes to zero; this follows from the Continuity of inner product (Yuksel). Indeed xmn,xmnxm~,xm~=xmn,xmnxmn,xm~+xmn,xm~xm~,xm~xmn,xmnxmn,xm~+xmn,xm~xm~,xm~=xmn,mnm~+mnm~,xm~xmnmnm~+mnm~xm~\begin{align*} \left| \langle x-m_{n}, x-m_{n} \rangle -\langle x-\tilde{m}, x-\tilde{m} \rangle \right| &= \left| \langle x-m_{n}, x-m_{n} \rangle -\langle x-m_{n}, x-\tilde{m} \rangle\right.\\ &\quad\quad\quad\left.+ \langle x-m_{n}, x-\tilde{m} \rangle- \langle x-\tilde{m}, x-\tilde{m} \rangle \right| \\ &\le \left| \langle x-m_{n}, x-m_{n} \rangle -\langle x-m_{n}, x-\tilde{m} \rangle \right| \\ &\quad\quad+ \left| \langle x-m_{n}, x-\tilde{m} \rangle -\langle x-\tilde{m}, x-\tilde{m} \rangle \right| \\ &= \left| \langle x-m_{n}, m_{n}-\tilde{m} \rangle \right| +\left| \langle m_{n}-\tilde{m}, x-\tilde{m} \rangle \right| \\ &\le \lVert x-m_{n} \rVert \lVert m_{n} -\tilde{m}\rVert +\lVert m_{n}-\tilde{m} \rVert \lVert x-\tilde{m} \rVert \end{align*}where the final inequality is due to Cauchy-Schwartz. Now, since xmnδ\lVert x-m_{n} \rVert\to\delta, we have that xmn\lVert x-m_{n} \rVert is bounded. Finally as mnm~0\lVert m_{n}-\tilde{m} \rVert\to0, both terms in the final line go to zero and we conclude that δ=limnxmn=xm~\delta=\lim_{ n \to \infty } \lVert x-m_{n} \rVert =\lVert x-\tilde{m} \rVert and therefore m~\tilde{m} is a minimizing vector. By part 1, this has to be the only minimizing vector in BB.

\end{proof}

Remark

So this is the statement of the theorem in the class notes but I’m ngl, it’s unnecessarily abstract. Pretty much what this says that is that WLOG imagine we’re working in 2D space HH and we take a closed subset BB which we can say lives along some line in HH. This theorem says that if we take any point bHb\in H then the closest point from x0Mx_{0}\in M to bb is one such that the difference vector bx0b-x_{0} is orthogonal to MM: 300 The projection here is the point x0x_{0}. This is our best approximation to bb; we’re essentially “projecting” bb to some lower fidelity space MM and x0x_{0} is chosen as the candidate projection since it best approximates bb with respect to the Metric (or Norm or in our case, Inner Product).

Theorem (2.3.1)

Let {ei}\{ e_{i} \} be a sequence of orthonormal vectors in a Hilbert space HH. Let {xn=i=1nϵiei}nN\left\{ x_{n}=\sum_{i=1}^{n}\epsilon_{i}e_{i} \right\}_{n\in \mathbb{N}}be a sequence of vectors in HH. The sequence converges to some vector xˉH\bar{x}\in H if and only if i=1ϵi2<\sum_{i=1}^{\infty}\left| \epsilon_{i} \right| ^{2}<\infty In this case, we have that xˉ,ei=ϵi\langle \bar{x}, e_{i} \rangle=\epsilon_{i}.

Theorem (2.3.3a)

For any two orthogonal vectors v,wv,w we have v+w2=v2+w2\lVert v+w \rVert ^{2}=\lVert v \rVert ^{2}+\lVert w \rVert ^{2}

Theorem (2.3.3)

Let HH be a separable Hilbert space. Then, every orthonormal system of vectors in HH has a finite or countably infinite number of elements.

\begin{proof} Let D={x1,x2,,}D=\{ x_{1},x_{2},\dots ,\} be a countable set dense in HH. Let {eα}αAH\{ e_{\alpha} \}_{\alpha \in \mathcal{A}}\subset H be a set of orthonormal vectors. Observe that by , for αβ\alpha \neq\beta:eαeβ2=eα2+eβ2=2\lVert e_{\alpha}-e_{\beta} \rVert^{2}=\lVert e_{\alpha} \rVert^{2}+\lVert e_{\beta} \rVert^{2} =2 hence eαeβ=2\lVert e_{\alpha}-e_{\beta} \rVert =\sqrt{ 2 } By denseness of DD, for every eαe_{\alpha}, there exists some element xkαDx_{k_{\alpha}}\in D, such that eαxkα<12.\lVert e_{\alpha}-x_{k_{\alpha}} \rVert < \frac{1}{\sqrt{ 2 }}.this is because we want to select xkαx_{k_{\alpha}} such that it is injective wrt eαe_{\alpha} and hence we need to restrict the radius using the following relation 2ϵ<2    ϵ<12.2\epsilon< \sqrt{ 2 }\iff\epsilon< \frac{1}{\sqrt{ 2 }}. Let MM be the collection of such vectors {xkα}αAD\{ x_{k_{\alpha}} \}_{\alpha \in \mathcal{A}}\subset D and observe that since it is a subset of DD, this set is countable as well. Now, for a given xkαx_{k_{\alpha}}, for any eβe_{\beta} with βα\beta \neq\alpha, we have, by the relation, xyxy\lVert x \rVert-\lVert y \rVert\le \lVert x-y \rVert eβeα(eαxkα)eβeα((eαxkα))=eβxkα\lVert e_{\beta}-e_{\alpha} \rVert -\lVert -(e_{\alpha}-x_{k_{\alpha}}) \rVert \le \lVert e_{\beta}-e_{\alpha}-(-(e_{\alpha}-x_{k_{\alpha}})) \rVert =\lVert e_{\beta}-x_{k_{\alpha}} \rVert and thus 212=12eβxkα\sqrt{ 2 }- \frac{1}{\sqrt{ 2 }}= \frac{1}{\sqrt{ 2 }}\le \lVert e_{\beta}-x_{k_{\alpha}} \rVert Therefore, for every xkαM,!eα:d(xkα,eα)<12x_{k_{\alpha}}\in M, !\exists e_{\alpha}:d(x_{k_{\alpha}},e_{\alpha})< \frac{1}{\sqrt{ 2 }}. Thus, we can associate with every element in {eα}αA\{ e_{\alpha} \}_{\alpha \in \mathcal{A}} a unique element in the countable set MDM\subset D. Thus, the set {eα}αA\{ e_{\alpha} \}_{\alpha \in \mathcal{A}} is countable. \end{proof} >[!def|2.3.5] Complete orthonormal sequence >An orthonormal sequence in a Hilbert space HH is complete if the only vector in HH which is orthogonal to each of the vectors is the null vector.

Lemma (2.3.4)

Let HH be a Hilbert space and let {ei}iN\{ e_{i} \}_{i\in \mathbb{N}} be an orthonormal sequence in HH. Then for any vector xHx\in H i=1x,ei2x2\sum_{i=1}^{\infty}|\langle x, e_{i} \rangle |^{2}\le \lVert x \rVert ^{2}

\begin{proof} For nNn\in \mathbb{N} let xn=i=1nx,eiei.x_{n}=\sum_{i=1}^{n}\langle x, e_{i} \rangle e_{i}.This vector xnx_{n} is the projection of xx onto the subspace spanned by the first nn basis vectors. The vector xxnx-x_{n} is orthogonal to this subspace, and thus orthogonal to xnx_{n} itself. By the we have x2=xn2+xxn2.\lVert x \rVert ^{2}=\lVert x_{n} \rVert ^{2}+\lVert x-x_{n} \rVert ^{2}.The proof proceeds by computing xxn2\lVert x-x_{n} \rVert^{2}: 0xi=1nx,eiei2=xi=1nx,eiei,xj=1nx,ejej=x2i=1nx,eix,eij=1nx,ejej,x+i=1nj=1nx,eix,ejei,ej=x2i=1nx,ei2j=1nx,ej2+i=1nx,ei2=x2i=1nx,ei2.\begin{align*} 0&\le \left\lVert x-\sum_{i=1}^{n}\langle x, e_{i} \rangle e_{i} \right\rVert ^{2}\\ &= \left\langle x-\sum_{i=1}^{n}\langle x, e_{i} \rangle e_{i}, x-\sum_{j=1}^{n}\langle x, e_{j} \rangle e_{j} \right\rangle \\ &= \lVert x \rVert ^{2}-\sum_{i=1}^{n} \overline{\langle x, e_{i} \rangle }\langle x, e_{i} \rangle -\sum_{j=1}^{n}\langle x, e_{j} \rangle \langle e_{j}, x \rangle +\sum_{i=1}^{n}\sum_{j=1}^{n}\overline{\langle x, e_{i} \rangle }\langle x, e_{j} \rangle \langle e_{i}, e_{j} \rangle \\ &= \lVert x \rVert ^{2}-\sum_{i=1}^{n}\left| \langle x, e_{i} \rangle \right| ^{2}-\sum_{j=1}^{n}\left| \langle x, e_{j} \rangle \right| ^{2}+\sum_{i=1}^{n}\left| \langle x, e_{i} \rangle \right| ^{2}\\ &= \lVert x \rVert ^{2}-\sum_{i=1}^{n}\left| \langle x, e_{i} \rangle \right| ^{2} .\end{align*}This holds for any n1n\ge 1. Since the partial sums are non-negative and bounded above by x2\lVert x \rVert^{2}, the series i=1x,ei2\sum_{i=1}^{\infty}\left| \langle x, e_{i} \rangle \right|^{2} converges and its sum is less than or equal to x2\lVert x \rVert^{2}. \end{proof}

Theorem (2.3.4)

A Hilbert space contains a complete orthonormal sequence if and only if it is separable.

\begin{proof}     \impliedby: Let HH be separable. Then, there exists a countable dense subset D={x1,x2,}D=\{ x_{1},x_{2},\dots \}. Apply the Gran-Schmidt process (212) to obtain {e1,e2,,}\{ e_{1},e_{2},\dots, \}, an orthonormal basis. We claim that this set is a complete orthonormal sequence.

Suppose not; that is, let hHh\in H be so that h0\lVert h \rVert\neq 0 and let h,ek=0,kN.(⭐®)\langle h, e_{k} \rangle=0,\forall k\in \mathbb{N}\tag{⭐️}.i.e., let hh be orthogonal to each eie_{i} and not the null vector. Now, for every ϵ>0\epsilon>0, there exists xmDx_{m}\in D with hxmϵ(3)\lVert h-x_{m} \rVert\le \epsilon\tag{3} and observe that xm=k=1mαiei(4)x_{m}=\sum_{k=1}^{m}\alpha_{i}e_{i}\tag{4}since the span of vectors {e1,,em}\{ e_{1},\dots,e_{m} \} contains xmx_{m}. Then, h2=h,h=hk=1mαiei,h=hxm,hhxmhϵh\lVert h \rVert ^{2}=\langle h, h \rangle =\left\langle h-\sum_{k=1}^{m}\alpha_{i}e_{i}, h \right\rangle =\langle h-x_{m}, h \rangle \le \lVert h-x_{m} \rVert \lVert h \rVert \le \epsilon \lVert h \rVert which is pretty much: Inner product defines a norm (Yuksel), add subtract summation term and apply linearity of inner product then apply (®)(⭐️), (4)(4), Cauchy-Schwarz Inequality (Yuksel), then (3)(3). This implies that hϵ\lVert h \rVert\le \epsilon. Since ϵ>0\epsilon>0 is arbitrary; then we have h=0\lVert h \rVert=0, a contradiction hence h=0\lVert h \rVert=0 and hh is the null element.

    \implies Now, let HH have a complete orthonormal sequence {e1,e2,}\{ e_{1},e_{2},\dots \}. We will show that D=nN{xH:x=i=1nαiei,αiQ},D=\bigcup_{n\in \mathbb{N}}\left\{ x\in H: x=\sum_{i=1}^{n}\alpha_{i}e_{i},\alpha_{i}\in \mathbb{Q} \right\},is a

  1. countable
  2. dense subset in HH, and separability of HH follows by the definition.
  1. That DD is countable follows from the fact that for every nn, the set {xH:x=i=1nαiei,αiQ}\left\{ x\in H: x=\sum_{i=1}^{n}\alpha_{i}e_{i},\alpha_{i}\in \mathbb{Q} \right\} is countable as a Cartesian product of finitely many countable sets, and thus the countable union over nNn\in \mathbb{N} leads to a countable set.

  2. We now show that this set is dense in HH. i.e., that xH,ϵ>0,eD:xe<ϵ\forall x\in H, \forall\epsilon>0, \exists e\in D:\lVert x-e\rVert<\epsilon where e=i=1Mαiei,αiQe=\sum_{i=1}^{M}\alpha_{i}e_{i},\alpha_{i}\in \mathbb{Q}.

Consider the sequence of vectors xn:=i=1nx,eieix_{n}:=\sum_{i=1}^{n}\langle x, e_{i} \rangle e_{i}. Observe that xnx_{n} converges to a limit by Theorem 2.3.1. To see that the sequence i=1x,ei2\sum_{i=1}^{\infty}\left| \langle x, e_{i} \rangle \right|^{2} is summable, use to conclude i=1x,ei2x2.\sum_{i=1}^{\infty}\left| \langle x, e_{i} \rangle \right| ^{2}\le \lVert x \rVert ^{2}.Therefore, the limit i=1x,eiei\sum_{i=1}^{\infty}\langle x, e_{i} \rangle e_{i} is well-defined and hence xnxx_{n}\to x for some xHx\in H.

Now, let h=xi=1x,eieih=x-\sum_{i=1}^{\infty} \langle x, e_{i} \rangle e_{i}. we claim that h=0\lVert h \rVert=0. For any mNm\in \mathbb{N}, h,em=xi=1x,eiei,em=x,emlimni=1nx,eiei,em=x,emlimni=1nx,eiei,em=x,emx,em=0\begin{align*}\langle h, e_{m} \rangle &= \left\langle x-\sum_{i=1}^{\infty}\langle x, e_{i} \rangle e_{i}, e_{m} \right\rangle \\ &= \langle x, e_{m} \rangle -\left\langle \lim_{ n \to \infty } \sum_{i=1}^{n}\langle x, e_{i} \rangle e_{i}, e_{m} \right\rangle \\ &= \langle x, e_{m} \rangle -\lim_{ n \to \infty } \left\langle \sum_{i=1}^{n}\langle x, e_{i} \rangle e_{i}, e_{m} \right\rangle \\ &= \langle x, e_{m} \rangle -\langle x, e_{m} \rangle \\ &= 0 \end{align*} But since {e1,e2,}\{ e_{1},e_{2},\dots \} is a complete orthonormal sequence, it must be that h=0\lVert h \rVert=0. Hence, x=i=1x,eieix=\sum_{i=1}^{\infty}\langle x, e_{i} \rangle e_{i}. Now approximate this vector, by first truncating the sum so that ϵ>0\forall\epsilon>0, xi=1Nx,eieiϵ2\left\lVert x-\sum_{i=1}^{N}\langle x, e_{i} \rangle e_{i} \right\rVert \le \frac{\epsilon}{2}and then approximating the coefficients by rational numbers so that i=1Nx,eieii=1Nαieiϵ2,αiQ.\left\lVert \sum_{i=1}^{N}\langle x, e_{i} \rangle e_{i}-\sum_{i=1}^{N}\alpha_{i}e_{i} \right\rVert \le \frac{\epsilon}{2},\quad\alpha_{i}\in \mathbb{Q}.This implies xi=1Nαieixi=1Nx,eiei+i=1Nx,eieii=1Nαieiϵ\begin{align*} \left\lVert x-\sum_{i=1}^{N}\alpha_{i}e_{i} \right\rVert &\le \left\lVert x-\sum_{i=1}^{N}\langle x, e_{i} \rangle e_{i} \right\rVert +\left\lVert \sum_{i=1}^{N}\langle x, e_{i} \rangle e_{i}-\sum_{i=1}^{N}\alpha_{i}e_{i} \right\rVert \\ &\le \epsilon \end{align*}Since for every ϵ\epsilon such an approximation can be made by some element in DD, this completes the proof.

\end{proof}

Theorem (2.3.5)

In a Hilbert space HH, a complete orthonormal sequence {en}nN\{ e_{n} \}_{n\in \mathbb{N}} defines a basis in the sense that for any xHx\in H, we have x=limNk=1Nx,ekekx=\lim_{ N \to \infty } \sum_{k=1}^{N}\langle x, e_{k} \rangle e_{k}

Linked from