v2009.01.01 - Convex Optimization

642 APPENDIX E. PROJECTION
For any shape of matrix A of any rank, and given any vector b that may
or may not be in R(A) , we wish to find a best Euclidean solution x ⋆ to
Ax = b (1754)
(more generally, Ax ≈ b given arbitrary matrices) by solving
minimize
x
‖Ax − b‖ 2 (1755)
Necessary and sufficient condition for optimal solution to this unconstrained
optimization is the so-called normal equation that results from zeroing the
convex objective’s gradient: (D.2.1)
A T Ax = A T b (1756)
normal because error vector b −Ax is perpendicular to R(A) ; id est,
A T (b −Ax)=0. Given any matrix A and any vector b , the normal equation
is solvable exactly; always so, because R(A T A)= R(A T ) and A T b∈ R(A T ).
When A is skinny-or-square full-rank, normal equation (1756) can be
solved exactly by inversion:
x ⋆ = (A T A) −1 A T b ≡ A † b (1757)
For matrix A of arbitrary rank and shape, on the other hand, A T A might
not be invertible. Yet the normal equation can always be solved exactly by:
(1745)
x ⋆ = lim A + t I) −1 A T b = A † b (1758)
t→0 +(AT
invertible for any positive value of t by (1354). The exact inversion (1757)
and this pseudoinverse solution (1758) each solve
lim minimize
t→0 + x
‖Ax − b‖ 2 + t ‖x‖ 2 (1759)
simultaneously providing least squares solution to (1755) and the classical
least norm solution E.1 [287, App.A.4] (confer3.1.3.0.1)
arg minimize ‖x‖ 2
x
subject to Ax = AA † b
(1760)
E.1 This means: optimal solutions of lesser norm than the so-called least norm solution
(1760) can be obtained (at expense of approximation Ax ≈ b hence, of perpendicularity)
by ignoring the limiting operation and introducing finite positive values of t into (1759).