The Gram-Schmidt Procedure and Orthogonal Projections 1 ...

Show that proj v (u) is the orthogonal projection of the vector u on the linear subspace spannedby the vector v. (Hint: show that the “error” u − proj v (u) is orthogonal to v, and hence isorthogonal to every vector in the linear subspace spanned by v.)In Matlab or Octave, the inner product between vectors u and v can be computed usingdot(u, v). Write a function (saving it in the file ‘PROJ.m’)function w = PROJ(u,v)that implements the projection operator proj v . (Caution: in Matlab, dot works correctlyfor real-valued and for complex-valued vectors, but in Octave, dot is incorrectly implementedfor complex-valued vectors. Throughout this lab, we will only use real-valuedvectors in R n .)Test that your function PROJ works correctly by computing the following projections:PROJ([1 2 3],[1 1 1]) =PROJ([1 2 3],[0 0 0]) =PROJ([1 2 3],[1 1 -1]) =The Gram-Schmidt procedure takes in a finite set of vectors {v 1 , . . . , v n } (not all ⃗0) in aninner-product space V and produces an orthonormal set {w 1 , . . . , w m }, m ≤ n, such thatL({w 1 , . . . , w m }) = L({v 1 , . . . , v n }), where L(·) denotes the linear span, i.e., the subspaceobtained from all possible linear combinations of the elements of the corresponding set.In a “naive” implementation, the Gram-Schmidt procedure would compute as follows(keeping only the nonzero w i ’s that are produced):u 1 = v 1 w 1 = u 1 /‖u 1 ‖ (or ⃗0 if u 1 = ⃗0)u 2 = v 2 − proj w1(v 2 ) w 2 = u 2 /‖u 2 ‖ (or ⃗0 if u 2 = ⃗0)u 3 = v 3 − proj w1(v 3 ) − proj w2(v 3 ) w 3 = u 3 /‖u 3 ‖ (or ⃗0 if u 3 = ⃗0).u i = v i − ∑ i−1k=1 proj w k(v i ) w i = u i /‖u i ‖.(or ⃗0 if u i = ⃗0)2

Write a function (saving it in the file ‘NAIVEGS.m’)function w = NAIVEGS(v)that implements the naive Gram-Schmidt procedure. The argument to the function is ann × l matrix v whose rows are vectors in R l . The output is an m × l matrix w whose rowsare the desired orthonormal basis.Programming notes (in Matlab or Octave):1. The number of rows of a matrix v can be obtained as size(v,1).2. The ith row of a matrix v can be addressed as v(i, :).3. The function zeros(size(x)) returns an all-zero array with the same shape asthat of x.4. A row can be appended to a matrix simply by assigning to it, e.g.,w(i,:)= u(j,:)assigns the jth row of u to the ith row of w.Test your function by computing the following.A = NAIVEGS([1 1; 1 0])B = NAIVEGS([0 0 0 0 ; 1 1 1 1])C = NAIVEGS([1 1 1 1; 1 1 0 0 ; 1 0 1 0; 1 0 0 0 ])To verify that the resulting basis is in fact orthonormal, compute the inner product of eachrow of the basis with every other row of the basis. This is conveniently done in Matlab orOctave by computing, say, C ∗ C ′ where C is a basis matrix. (Here C ′ denotes the transposeof the matrix C.)If C is an m × l matrix whose rows form an orthonormal basis, what matrix shouldresult from the computation C ∗ C ′ ?3

Unfortunately, the naive Gram-Schmidt procedure is numerically unstable, which meansthat it is highly sensitive to numerical errors. For example, try running NAIVEGS on the7th order Hilbert matrix:H7 = [ 1/1 1/2 1/3 1/4 1/5 1/6 1/7;1/2 1/3 1/4 1/5 1/6 1/7 1/8;1/3 1/4 1/5 1/6 1/7 1/8 1/9;1/4 1/5 1/6 1/7 1/8 1/9 1/10;1/5 1/6 1/7 1/8 1/9 1/10 1/11;1/6 1/7 1/8 1/9 1/10 1/11 1/12;1/7 1/8 1/9 1/10 1/11 1/12 1/13 ];C = NAIVEGS(H7);Z = abs(C * C’ - eye(7))In this computation, eye(7) denotes the 7 × 7 identity matrix, and abs computes theabsolute value of each matrix component.Ideally, what should Z be?What is the actual largest entry in Z?Fortunately, the Gram-Schmidt procedure can be made numerically stable via a simplemodification. Instead of computing∑i−1u i = v i − proj wk(v i )as in the naive Gram-Schmidt algorithm, compute u i in a series of steps:k=1u (1)i = v i − proj w1(v i )u (2)i = u (1)i − proj w2(u (1)i )u (3)i = u (2)i − proj w3(u (2)i ). .u i = u (i−2)i − proj wi−1(u (i−2)i )Copy your file NAIVEGS.m to the file GS.m and modify GS.m to implement the modifiedGram-Schmidt procedure as the functionfunction w = GS(v)4

Using your new modified GS procedure, try running:D = GS(H7);Z = abs(D * D’ - eye(7))Now what is the largest entry in Z?3 Orthogonal ProjectionsLet P be a finite-dimensional subspace of an inner-product space V . Once an orthonormalbasis {w 1 , w 2 , . . . , w m } for P is obtained, the orthogonal projection of an arbitrary vectorv ∈ V on P is easily computed asˆv =m∑〈v, w i 〉w i .i=1The vector ˆv is the orthogonal projection of v on P and the components of the m-tuple(〈v, w 1 〉, 〈v, w 2 〉, . . . , 〈v, w m 〉)are the coordinates of ˆv with respect to the given orthonormal basis.Let B be an m × l matrix whose rows form an orthonormal basis for subspace P . (Forexample, B might be produced by your GS function.) Let v be an arbitrary 1 × l rowvector, and let ˆv be the orthogonal projection of v on P .• Give a Matlab or Octave expression (in terms of B and v) for the coordinatesof ˆv with respect to basis given in the rows of B.• Give a Matlab or Octave expression (in terms of B and v) for ˆv.5

For example, let P be the subspace of R 3 consisting of all vectors (x, y, z) with theproperty that x + y + z = 0. Find an orthonormal basis for P and compute theorthogonal projection of the following vectors on P .• v = [1,2,3], ˆv =• v = [1,1,1], ˆv =• v = [0,1,0], ˆv =In the above computations, let e denote the “error vector” v − ˆv. Compute 〈v, v〉, 〈ˆv, ˆv〉and 〈e, e〉 for the three cases above.Write a general relationship among the three quantities 〈v, v〉, 〈ˆv, ˆv〉 and 〈e, e〉, wheree = v − ˆv.4 The QR DecompositionLet V be a square matrix (assumed invertible) and let Q = GS(V) be the orthogonal matrixreturned by your Gram-Schmidt procedure. Each row of V has a set of coordinates withrespect to the rows of Q. This implies thatfor some matrix R.V = RQ,After trying a few examples, explain what special property the matrix R has.random n × n matrix can be obtained in Matlab or Octave via rand(n, n).A6

In linear algebra, this decomposition is usually written in transposed form, expressing aninvertible matrix V as V = QR where Q is an orthogonal matrix and R has a similar specialproperty as above. Such a decomposition is referred to as the “QR” decomposition.Thus we see that the Gram-Schmidt procedure results in a QR decomposition. By typing’help qr’, investigate whether your version of Matlab or Octave implements the QRdecomposition.5 Approximate Curve FittingConsider the approximation of a function f(x) over the interval [0, 1] as a quadratic polynomial.Here we show how this can be done approximately using orthogonal projections.In Matlab or Octave, run the following:N = 9;a = linspace(0,1,N);v(1,:) = ones(size(a));v(2,:) = a;v(3,:) = a .* a;b = GS(v);The linspace(a,b,N) function returns a vector of length N with samples uniformlyspaced from a to b, inclusive. The ‘.*’ operator takes the componentwise product of thecorresponding vectors.The matrix v now contains three rows: the first row contains samples from the constantfunction; the second row contains samples from a linearly increasing function; and thethird row contains samples from a quadratically increasing function.The matrix b contains a corresponding orthonormal basis. To see the corresponding samples,try the following command:plot(b(1,:)); hold; plot(b(2,:)); plot(b(3,:));Let us populate a vector f with samples from some function, e.g.,f = exp(a);Let g denote the orthogonal projection of f on the space spanned by the rows of v or b.Both f and g can be plotted as follows:7

hold off; plot(f); hold; plot(g);Express g as a linear combination of the rows of v. (N.B.: we are not asking for thecoordinates of g with respect to the orthonormal basis, but rather with respect to theoriginal basis.)A more accurate representation can be obtained by increasing the number of samples.Repeat this exercise for N = 100.When N = 100, the coordinates of g with respect to the rows of v are:8