This formulation allows us to develop approximations of signals using a finite number of basis signals
We can state a projection theorem which certifies optimality in a wide variety of applications.
Theorem (2.2.3)
Let H be a Hilbert space and B a subspace of H. Consider the problem of approximating some point x∈H using a point from B: m∈Binf∥x−m∥
A necessary and sufficient condition for m∗∈B to be the minimizing element in B so that m∈Binf∥x−m∥=∥x−m∗∥(1)or equivalently ∥x−m∗∥≤∥x−y∥∀y∈Bis that x−m∗ is orthogonal to B. If m∗ exists, then it is also unique.
Let H be a Hilbert space and B be a closedsubspace of H. For any vector x∈H, there is a unique vector m∗∈B satisfying (1).
\begin{proof} We first show 1. Let x∈H, m0∈B. Want to show that if x−m0 is not Orthogonal vector to B then it cannot be optimal.
Suppose m0∈B is such that ∃m∈B with ⟨x−m0,m⟩>0, i.e., x−m0 is not orthogonal to B.
WLOG, take ∥m∥=1 and ⟨x−m0,m⟩=δ.
We can show that with m1=m0+δm, we will have ∥x−m1∥2=⟨x−m1,x−m1⟩2=⟨x−m0−δm,x−m0−δm⟩=⟨x−m0,x−m0⟩−δ⟨m,x−m0⟩−δ⟨x−m0,m⟩+δ2⟨m,m⟩=∥x−m0∥2−2δ2+δ2=∥x−m0∥2−δ2<∥x−m0∥2and thus m0 cannot be a minimizer of (1).
We now show that if x−m∗ is orthogonal to B, then it is optimal. If ⟨x−m∗,M⟩=0, we have that for any m∈B ∥x−m∥2=∥x−m∗+m∗−m∥2=∥x−m∗∥2+∥m∗−m∥2≥∥x−m∗∥2 since for x⊥y we have that ∥x+y∥2=∥x∥2+∥y∥2. This yields a contradiction, proving 1.
We now show 2. Let δ=m∈Binf∥x−m∥Let {mk}k∈N be s.t. ∥x−mk∥→δand observe that ⟨x−mk+x−mn,x−mk+x−mn⟩+⟨x−mk−(x−mn),x−mk−(x−mn)⟩=⟨x−mk,x−mk+x−mn⟩+⟨x−mn,x−mk+x−mn⟩+⟨x−mk,x−mk−(x−mn)⟩+⟨−(x−mn),x−mk−(x−mn)⟩=2⟨x−mk,x−mk⟩+2⟨x−mn,x−mn⟩Write ⟨x−mk+x−mn,x−mk+x−mn⟩=⟨2(x−2mk+mn),2(x−2mk+mn)⟩=4⟨x−2mk+mn,x−2mk+mn⟩(🚧)Since 2mk+mn∈B then by our definition of δ we have that: x−2mk+mn≥δthen we square both sides then multiply by 44x−2mk+mn2≥4δ2 and since the LHS is exactly (🚧) then by applying Parallelogram law we get ∥mk−mn∥2≤2∥x−mn∥2+2∥x−mk∥2−4δ2As a result, as ∥x−mn∥2→δ2, we have that mk is Cauchy. Since B is closed, it has a limit, call the limit m~. We claim that the limit is optimal and hence m~=m∗. Conside the difference: ⟨x−mn,x−mn⟩−⟨x−m~,x−m~⟩ We claim that the difference goes to zero; this follows from the Continuity of inner product (Yuksel). Indeed ∣⟨x−mn,x−mn⟩−⟨x−m~,x−m~⟩∣=∣⟨x−mn,x−mn⟩−⟨x−mn,x−m~⟩+⟨x−mn,x−m~⟩−⟨x−m~,x−m~⟩∣≤∣⟨x−mn,x−mn⟩−⟨x−mn,x−m~⟩∣+∣⟨x−mn,x−m~⟩−⟨x−m~,x−m~⟩∣=∣⟨x−mn,mn−m~⟩∣+∣⟨mn−m~,x−m~⟩∣≤∥x−mn∥∥mn−m~∥+∥mn−m~∥∥x−m~∥where the final inequality is due to Cauchy-Schwartz. Now, since ∥x−mn∥→δ, we have that ∥x−mn∥ is bounded. Finally as ∥mn−m~∥→0, both terms in the final line go to zero and we conclude that δ=n→∞lim∥x−mn∥=∥x−m~∥and therefore m~ is a minimizing vector. By part 1, this has to be the only minimizing vector in B.
\end{proof}
Remark
So this is the statement of the theorem in the class notes but I’m ngl, it’s unnecessarily abstract. Pretty much what this says that is that WLOG imagine we’re working in 2D space H and we take a closed subset B which we can say lives along some line in H. This theorem says that if we take any point b∈H then the closest point from x0∈M to b is one such that the difference vector b−x0 is orthogonal to M: The projection here is the point x0. This is our best approximation to b; we’re essentially “projecting” b to some lower fidelity space M and x0 is chosen as the candidate projection since it best approximates b with respect to the Metric (or Norm or in our case, Inner Product).
Theorem (2.3.1)
Let {ei} be a sequence of orthonormal vectors in a Hilbert spaceH. Let {xn=i=1∑nϵiei}n∈Nbe a sequence of vectors in H. The sequence converges to some vector xˉ∈H if and only if i=1∑∞∣ϵi∣2<∞ In this case, we have that ⟨xˉ,ei⟩=ϵi.
\begin{proof} Let D={x1,x2,…,} be a countable set dense in H. Let {eα}α∈A⊂H be a set of orthonormal vectors. Observe that by , for α=β:∥eα−eβ∥2=∥eα∥2+∥eβ∥2=2hence ∥eα−eβ∥=2 By denseness of D, for every eα, there exists some element xkα∈D, such that ∥eα−xkα∥<21.this is because we want to select xkα such that it is injective wrt eα and hence we need to restrict the radius using the following relation 2ϵ<2⟺ϵ<21. Let M be the collection of such vectors {xkα}α∈A⊂D and observe that since it is a subset of D, this set is countable as well. Now, for a given xkα, for any eβ with β=α, we have, by the relation, ∥x∥−∥y∥≤∥x−y∥∥eβ−eα∥−∥−(eα−xkα)∥≤∥eβ−eα−(−(eα−xkα))∥=∥eβ−xkα∥and thus 2−21=21≤∥eβ−xkα∥Therefore, for every xkα∈M,!∃eα:d(xkα,eα)<21. Thus, we can associate with every element in {eα}α∈A a unique element in the countable set M⊂D. Thus, the set {eα}α∈A is countable. \end{proof} >[!def|2.3.5] Complete orthonormal sequence >An orthonormal sequence in a Hilbert spaceH is complete if the only vector in H which is orthogonal to each of the vectors is the null vector.
Lemma (2.3.4)
Let H be a Hilbert space and let {ei}i∈N be an orthonormal sequence in H. Then for any vector x∈Hi=1∑∞∣⟨x,ei⟩∣2≤∥x∥2
\begin{proof} For n∈N let xn=i=1∑n⟨x,ei⟩ei.This vector xn is the projection of x onto the subspace spanned by the first n basis vectors. The vector x−xn is orthogonal to this subspace, and thus orthogonal to xn itself. By the we have ∥x∥2=∥xn∥2+∥x−xn∥2.The proof proceeds by computing ∥x−xn∥2: 0≤x−i=1∑n⟨x,ei⟩ei2=⟨x−i=1∑n⟨x,ei⟩ei,x−j=1∑n⟨x,ej⟩ej⟩=∥x∥2−i=1∑n⟨x,ei⟩⟨x,ei⟩−j=1∑n⟨x,ej⟩⟨ej,x⟩+i=1∑nj=1∑n⟨x,ei⟩⟨x,ej⟩⟨ei,ej⟩=∥x∥2−i=1∑n∣⟨x,ei⟩∣2−j=1∑n∣⟨x,ej⟩∣2+i=1∑n∣⟨x,ei⟩∣2=∥x∥2−i=1∑n∣⟨x,ei⟩∣2.This holds for any n≥1. Since the partial sums are non-negative and bounded above by ∥x∥2, the series ∑i=1∞∣⟨x,ei⟩∣2 converges and its sum is less than or equal to ∥x∥2. \end{proof}
\begin{proof}⟸: Let H be separable. Then, there exists a countable dense subset D={x1,x2,…}. Apply the Gran-Schmidt process (212) to obtain {e1,e2,…,}, an orthonormalbasis. We claim that this set is a complete orthonormal sequence.
Suppose not; that is, let h∈H be so that ∥h∥=0 and let ⟨h,ek⟩=0,∀k∈N.(⭐R◯)i.e., let h be orthogonal to each ei and not the null vector. Now, for every ϵ>0, there exists xm∈D with ∥h−xm∥≤ϵ(3) and observe that xm=k=1∑mαiei(4)since the span of vectors {e1,…,em} contains xm. Then, ∥h∥2=⟨h,h⟩=⟨h−k=1∑mαiei,h⟩=⟨h−xm,h⟩≤∥h−xm∥∥h∥≤ϵ∥h∥ which is pretty much: Inner product defines a norm (Yuksel), add subtract summation term and apply linearity of inner product then apply (⭐R◯), (4), Cauchy-Schwarz Inequality (Yuksel), then (3). This implies that ∥h∥≤ϵ. Since ϵ>0 is arbitrary; then we have ∥h∥=0, a contradiction hence ∥h∥=0 and h is the null element.
⟹ Now, let H have a complete orthonormal sequence{e1,e2,…}. We will show that D=n∈N⋃{x∈H:x=i=1∑nαiei,αi∈Q},is a
countable
dense subset in H, and separability of H follows by the definition.
That D is countable follows from the fact that for every n, the set {x∈H:x=∑i=1nαiei,αi∈Q} is countable as a Cartesian product of finitely many countable sets, and thus the countable union over n∈N leads to a countable set.
We now show that this set is dense in H. i.e., that ∀x∈H,∀ϵ>0,∃e∈D:∥x−e∥<ϵ where e=∑i=1Mαiei,αi∈Q.
Consider the sequence of vectors xn:=∑i=1n⟨x,ei⟩ei. Observe that xn converges to a limit by Theorem 2.3.1. To see that the sequence ∑i=1∞∣⟨x,ei⟩∣2 is summable, use to conclude i=1∑∞∣⟨x,ei⟩∣2≤∥x∥2.Therefore, the limit ∑i=1∞⟨x,ei⟩ei is well-defined and hence xn→x for some x∈H.
Now, let h=x−∑i=1∞⟨x,ei⟩ei. we claim that ∥h∥=0. For any m∈N, ⟨h,em⟩=⟨x−i=1∑∞⟨x,ei⟩ei,em⟩=⟨x,em⟩−⟨n→∞limi=1∑n⟨x,ei⟩ei,em⟩=⟨x,em⟩−n→∞lim⟨i=1∑n⟨x,ei⟩ei,em⟩=⟨x,em⟩−⟨x,em⟩=0 But since {e1,e2,…} is a complete orthonormal sequence, it must be that ∥h∥=0. Hence, x=∑i=1∞⟨x,ei⟩ei. Now approximate this vector, by first truncating the sum so that ∀ϵ>0, x−i=1∑N⟨x,ei⟩ei≤2ϵand then approximating the coefficients by rational numbers so that i=1∑N⟨x,ei⟩ei−i=1∑Nαiei≤2ϵ,αi∈Q.This implies x−i=1∑Nαiei≤x−i=1∑N⟨x,ei⟩ei+i=1∑N⟨x,ei⟩ei−i=1∑Nαiei≤ϵSince for every ϵ such an approximation can be made by some element in D, this completes the proof.