v2010.10.26 - Convex Optimization

702 APPENDIX E. PROJECTIONGiven arbitrary matrix A (of any rank and dimension) and vector b notnecessarily in R(A) , we wish to find a best solution x ⋆ toAx ≈ b (1878)in a Euclidean sense by solving an algebraic expression for orthogonalprojection of b on R(A)minimize ‖Ax − b‖ 2 (1879)xNecessary and sufficient condition for optimal solution to this unconstrainedoptimization is the so-called normal equation that results from zeroing theconvex objective’s gradient: (D.2.1)A T Ax = A T b (1880)normal because error vector b −Ax is perpendicular to R(A) ; id est,A T (b −Ax)=0. Given any matrix A and any vector b , the normal equationis solvable exactly; always so, because R(A T A)= R(A T ) and A T b∈ R(A T ).Given particular ˜x∈ R(A T ) solving (1880), then (Figure 161) it isnecessarily unique and ˜x = x ⋆ = A † b . When A is skinny-or-square full-rank,normal equation (1880) can be solved exactly by inversion:x ⋆ = (A T A) −1 A T b ≡ A † b (1881)For matrix A of arbitrary rank and shape, on the other hand, A T A mightnot be invertible. Yet the normal equation can always be solved exactly by:(1869)x ⋆ = lim A + t I) −1 A T b = A † b (1882)t→0 +(ATinvertible for any positive value of t by (1451). The exact inversion (1881)and this pseudoinverse solution (1882) each solve a limited regularization∥ [ ]∥b ∥∥∥20limt→0 +minimize ‖Ax − b‖ 2 + t ‖x‖ 2 ≡xlimt→0 +minimize x[ √t A∥]x − I(1883)simultaneously providing least squares solution to (1879) and the classicalleast norm solution E.1 [331, App.A.4] [48] (confer3.2.0.0.1)arg minimize ‖x‖ 2xsubject to Ax = AA † b(1884)E.1 This means: optimal solutions of lesser norm than the so-called least norm solution(1884) can be obtained (at expense of approximation Ax ≈ b hence, of perpendicularity)by ignoring the limiting operation and introducing finite positive values of t into (1883).