v2007.09.13 - Convex Optimization

581Equivalent to the corresponding Moore-Penrose condition:1. A T = A T AA † or A T = A † AA T2. A †T = A †T A † A or A †T = AA † A †TWhen A is symmetric, A † is symmetric and (A.6)A ≽ 0 ⇔ A † ≽ 0 (1641)E.0.1.0.1 Example. Solution to classical linear equation Ax = b .In2.5.1.1, the solution set to matrix equation Ax = b was representedas an intersection of hyperplanes. Regardless of rank of A or its shape(fat or skinny), interpretation as a hyperplane intersection describing apossibly empty affine set generally holds. If matrix A is rank deficient orfat, there is an infinity of solutions x when b∈R(A). A unique solutionoccurs when the hyperplanes intersect at a single point.For any shape of matrix A of any rank, and given any vector b that mayor may not be in R(A) , we wish to find a best Euclidean solution x ⋆ toAx = b (1642)(more generally, Ax ≈ b given arbitrary matrices) by solvingminimizex‖Ax − b‖ 2 (1643)Necessary and sufficient condition for optimal solution to this unconstrainedoptimization is the so-called normal equation that results from zeroing theconvex objective’s gradient: (D.2.1)A T Ax = A T b (1644)normal because error vector b −Ax is perpendicular to R(A) ; id est,A T (b −Ax)=0. Given any matrix A and any vector b , the normal equationis solvable exactly; always so, because R(A T A)= R(A T ) and A T b∈ R(A T ).When A is skinny-or-square full-rank, normal equation (1644) can besolved exactly by inversion:x ⋆ = (A T A) −1 A T b ≡ A † b (1645)