v2007.09.13 - Convex Optimization

582 APPENDIX E. PROJECTIONFor matrix A of arbitrary rank and shape, on the other hand, A T A mightnot be invertible. Yet the normal equation can always be solved exactly by:(1633)x ⋆ = lim A + t I) −1 A T b = A † b (1646)t→0 +(ATinvertible for any positive value of t by (1251). The exact inversion (1645)and this pseudoinverse solution (1646) each solvelim minimizet→0 + x‖Ax − b‖ 2 + t ‖x‖ 2 (1647)simultaneously providing least squares solution to (1643) and the classicalleast norm solution E.1 [247, App.A.4] (conferE.5.0.0.5)arg minimize ‖x‖ 2xsubject to Ax = AA † bwhere AA † b is the orthogonal projection of vector b on R(A).(1648)E.1 Idempotent matricesProjection matrices are square and defined by idempotence, P 2 =P ;[247,2.6] [151,1.3] equivalent to the condition, P be diagonalizable[149,3.3, prob.3] with eigenvalues φ i ∈ {0, 1}. [298,4.1, thm.4.1]Idempotent matrices are not necessarily symmetric. The transpose of anidempotent matrix remains idempotent; P T P T = P T . Solely exceptingP = I , all projection matrices are neither orthogonal (B.5) or invertible.[247,3.4] The collection of all projection matrices of particular dimensiondoes not form a convex set.Suppose we wish to project nonorthogonally (obliquely) on the rangeof any particular matrix A∈ R m×n . All idempotent matrices projectingnonorthogonally on R(A) may be expressed:P = A(A † + BZ T ) ∈ R m×m (1649)E.1 This means: optimal solutions of lesser norm than the so-called least norm solution(1648) can be obtained (at expense of approximation Ax ≈ b hence, of perpendicularity)by ignoring the limiting operation and introducing finite positive values of t into (1647).