v2009.01.01 - Convex Optimization

A.6. SINGULAR VALUE DECOMPOSITION, SVD 567
zero rows or n−η zero columns; the nonincreasingly ordered (possibly 0)
singular values appear along its main diagonal as for compact SVD (1453).
An important geometrical interpretation of SVD is given in Figure 131
for m = n = 2 : The image of the unit sphere under any m × n matrix
multiplication is an ellipse. Considering the three factors of the SVD
separately, note that Q T is a pure rotation of the circle. Figure 131 shows
how the axes q 1 and q 2 are first rotated by Q T to coincide with the coordinate
axes. Second, the circle is stretched by Σ in the directions of the coordinate
axes to form an ellipse. The third step rotates the ellipse by U into its
final position. Note how q 1 and q 2 are rotated to end up as u 1 and u 2 , the
principal axes of the final ellipse. A direct calculation shows that Aq j = σ j u j .
Thus q j is first rotated to coincide with the j th coordinate axis, stretched by
a factor σ j , and then rotated to point in the direction of u j . All of this
is beautifully illustrated for 2 ×2 matrices by the Matlab code eigshow.m
(see [290]).
A direct consequence of the geometric interpretation is that the largest
singular value σ 1 measures the “magnitude” of A (its 2-norm):
‖A‖ 2 = sup ‖Ax‖ 2 = σ 1 (1461)
‖x‖ 2 =1
This means that ‖A‖ 2 is the length of the longest principal semiaxis of the
ellipse.
Expressions for U , Q , and Σ follow readily from (1460),
AA T U = UΣΣ T and A T AQ = QΣ T Σ (1462)
demonstrating that the columns of U are the eigenvectors of AA T and the
columns of Q are the eigenvectors of A T A. −Neil Muller et alii [234]
A.6.4
Pseudoinverse by SVD
Matrix pseudoinverse (E) is nearly synonymous with singular value
decomposition because of the elegant expression, given A = UΣQ T
A † = QΣ †T U T ∈ R n×m (1463)
that applies to all three flavors of SVD, where Σ †
entries of matrix Σ .
simply inverts nonzero