v2007.09.13 - Convex Optimization
v2007.09.13 - Convex Optimization v2007.09.13 - Convex Optimization
582 APPENDIX E. PROJECTIONFor matrix A of arbitrary rank and shape, on the other hand, A T A mightnot be invertible. Yet the normal equation can always be solved exactly by:(1633)x ⋆ = lim A + t I) −1 A T b = A † b (1646)t→0 +(ATinvertible for any positive value of t by (1251). The exact inversion (1645)and this pseudoinverse solution (1646) each solvelim minimizet→0 + x‖Ax − b‖ 2 + t ‖x‖ 2 (1647)simultaneously providing least squares solution to (1643) and the classicalleast norm solution E.1 [247, App.A.4] (conferE.5.0.0.5)arg minimize ‖x‖ 2xsubject to Ax = AA † bwhere AA † b is the orthogonal projection of vector b on R(A).(1648)E.1 Idempotent matricesProjection matrices are square and defined by idempotence, P 2 =P ;[247,2.6] [151,1.3] equivalent to the condition, P be diagonalizable[149,3.3, prob.3] with eigenvalues φ i ∈ {0, 1}. [298,4.1, thm.4.1]Idempotent matrices are not necessarily symmetric. The transpose of anidempotent matrix remains idempotent; P T P T = P T . Solely exceptingP = I , all projection matrices are neither orthogonal (B.5) or invertible.[247,3.4] The collection of all projection matrices of particular dimensiondoes not form a convex set.Suppose we wish to project nonorthogonally (obliquely) on the rangeof any particular matrix A∈ R m×n . All idempotent matrices projectingnonorthogonally on R(A) may be expressed:P = A(A † + BZ T ) ∈ R m×m (1649)E.1 This means: optimal solutions of lesser norm than the so-called least norm solution(1648) can be obtained (at expense of approximation Ax ≈ b hence, of perpendicularity)by ignoring the limiting operation and introducing finite positive values of t into (1647).
E.1. IDEMPOTENT MATRICES 583where R(P )= R(A) , E.2 B ∈ R n×k for k ∈{1... m} is otherwise arbitrary,and Z ∈ R m×k is any matrix whose range is in N(A T ) ; id est,A T Z = A † Z = 0 (1650)Evidently, the collection of nonorthogonal projectors projecting on R(A) isan affine subsetP k = { A(A † + BZ T ) | B ∈ R n×k} (1651)When matrix A in (1649) is skinny full-rank (A † A = I) or has orthonormalcolumns (A T A = I), either property leads to a biorthogonal characterizationof nonorthogonal projection:E.1.1Biorthogonal characterizationAny nonorthogonal projector P 2 =P ∈ R m×m projecting on nontrivial R(U)can be defined by a biorthogonality condition Q T U =I ; the biorthogonaldecomposition of P being (confer (1649))where E.3 (B.1.1.1)and where generally (confer (1678)) E.4P = UQ T , Q T U = I (1652)R(P )= R(U) , N(P )= N(Q T ) (1653)P T ≠ P , P † ≠ P , ‖P ‖ 2 ≠ 1, P 0 (1654)and P is not nonexpansive (1679).E.2 Proof. R(P )⊆ R(A) is obvious [247,3.6]. By (119) and (120),R(A † + BZ T ) = {(A † + BZ T )y | y ∈ R m }⊇ {(A † + BZ T )y | y ∈ R(A)} = R(A T )R(P ) = {A(A † + BZ T )y | y ∈ R m }⊇ {A(A † + BZ T )y | (A † + BZ T )y ∈ R(A T )} = R(A)E.3 Proof. Obviously, R(P ) ⊆ R(U). Because Q T U = IR(P ) = {UQ T x | x ∈ R m }⊇ {UQ T Uy | y ∈ R k } = R(U)E.4 Orthonormal decomposition (1675) (conferE.3.4) is a special case of biorthogonaldecomposition (1652) characterized by (1678). So, these characteristics (1654) are notnecessary conditions for biorthogonality.
- Page 532 and 533: 532 APPENDIX B. SIMPLE MATRICESFigu
- Page 534 and 535: 534 APPENDIX B. SIMPLE MATRICESB.5.
- Page 536 and 537: 536 APPENDIX C. SOME ANALYTICAL OPT
- Page 538 and 539: 538 APPENDIX C. SOME ANALYTICAL OPT
- Page 540 and 541: 540 APPENDIX C. SOME ANALYTICAL OPT
- Page 542 and 543: 542 APPENDIX C. SOME ANALYTICAL OPT
- Page 544 and 545: 544 APPENDIX C. SOME ANALYTICAL OPT
- Page 546 and 547: 546 APPENDIX C. SOME ANALYTICAL OPT
- Page 548 and 549: 548 APPENDIX C. SOME ANALYTICAL OPT
- Page 550 and 551: 550 APPENDIX D. MATRIX CALCULUSThe
- Page 552 and 553: 552 APPENDIX D. MATRIX CALCULUSGrad
- Page 554 and 555: 554 APPENDIX D. MATRIX CALCULUSBeca
- Page 556 and 557: 556 APPENDIX D. MATRIX CALCULUSwhic
- Page 558 and 559: 558 APPENDIX D. MATRIX CALCULUS⎡
- Page 560 and 561: 560 APPENDIX D. MATRIX CALCULUS→Y
- Page 562 and 563: 562 APPENDIX D. MATRIX CALCULUSD.1.
- Page 564 and 565: 564 APPENDIX D. MATRIX CALCULUSwhic
- Page 566 and 567: 566 APPENDIX D. MATRIX CALCULUSIn t
- Page 568 and 569: 568 APPENDIX D. MATRIX CALCULUSD.1.
- Page 570 and 571: 570 APPENDIX D. MATRIX CALCULUSD.2
- Page 572 and 573: 572 APPENDIX D. MATRIX CALCULUSalge
- Page 574 and 575: 574 APPENDIX D. MATRIX CALCULUStrac
- Page 576 and 577: 576 APPENDIX D. MATRIX CALCULUSD.2.
- Page 578 and 579: 578 APPENDIX D. MATRIX CALCULUS
- Page 580 and 581: 580 APPENDIX E. PROJECTIONThe follo
- Page 584 and 585: 584 APPENDIX E. PROJECTION(⇐) To
- Page 586 and 587: 586 APPENDIX E. PROJECTIONNonorthog
- Page 588 and 589: 588 APPENDIX E. PROJECTIONE.2.0.0.1
- Page 590 and 591: 590 APPENDIX E. PROJECTIONE.3.2Orth
- Page 592 and 593: 592 APPENDIX E. PROJECTIONE.3.5Unif
- Page 594 and 595: 594 APPENDIX E. PROJECTIONE.4 Algeb
- Page 596 and 597: 596 APPENDIX E. PROJECTIONa ∗ 2K
- Page 598 and 599: 598 APPENDIX E. PROJECTIONwhere Y =
- Page 600 and 601: 600 APPENDIX E. PROJECTION(B.4.2).
- Page 602 and 603: 602 APPENDIX E. PROJECTIONis a nono
- Page 604 and 605: 604 APPENDIX E. PROJECTIONE.6.4.1Or
- Page 606 and 607: 606 APPENDIX E. PROJECTIONq i q T i
- Page 608 and 609: 608 APPENDIX E. PROJECTIONThe test
- Page 610 and 611: 610 APPENDIX E. PROJECTIONPerpendic
- Page 612 and 613: 612 APPENDIX E. PROJECTIONE.8 Range
- Page 614 and 615: 614 APPENDIX E. PROJECTIONAs for su
- Page 616 and 617: 616 APPENDIX E. PROJECTIONWith refe
- Page 618 and 619: 618 APPENDIX E. PROJECTIONProjectio
- Page 620 and 621: 620 APPENDIX E. PROJECTIONE.9.2.2.2
- Page 622 and 623: 622 APPENDIX E. PROJECTIONThe foreg
- Page 624 and 625: 624 APPENDIX E. PROJECTION❇❇❇
- Page 626 and 627: 626 APPENDIX E. PROJECTIONE.10 Alte
- Page 628 and 629: 628 APPENDIX E. PROJECTIONbH 1H 2P
- Page 630 and 631: 630 APPENDIX E. PROJECTIONa(a){y |
E.1. IDEMPOTENT MATRICES 583where R(P )= R(A) , E.2 B ∈ R n×k for k ∈{1... m} is otherwise arbitrary,and Z ∈ R m×k is any matrix whose range is in N(A T ) ; id est,A T Z = A † Z = 0 (1650)Evidently, the collection of nonorthogonal projectors projecting on R(A) isan affine subsetP k = { A(A † + BZ T ) | B ∈ R n×k} (1651)When matrix A in (1649) is skinny full-rank (A † A = I) or has orthonormalcolumns (A T A = I), either property leads to a biorthogonal characterizationof nonorthogonal projection:E.1.1Biorthogonal characterizationAny nonorthogonal projector P 2 =P ∈ R m×m projecting on nontrivial R(U)can be defined by a biorthogonality condition Q T U =I ; the biorthogonaldecomposition of P being (confer (1649))where E.3 (B.1.1.1)and where generally (confer (1678)) E.4P = UQ T , Q T U = I (1652)R(P )= R(U) , N(P )= N(Q T ) (1653)P T ≠ P , P † ≠ P , ‖P ‖ 2 ≠ 1, P 0 (1654)and P is not nonexpansive (1679).E.2 Proof. R(P )⊆ R(A) is obvious [247,3.6]. By (119) and (120),R(A † + BZ T ) = {(A † + BZ T )y | y ∈ R m }⊇ {(A † + BZ T )y | y ∈ R(A)} = R(A T )R(P ) = {A(A † + BZ T )y | y ∈ R m }⊇ {A(A † + BZ T )y | (A † + BZ T )y ∈ R(A T )} = R(A)E.3 Proof. Obviously, R(P ) ⊆ R(U). Because Q T U = IR(P ) = {UQ T x | x ∈ R m }⊇ {UQ T Uy | y ∈ R k } = R(U)E.4 Orthonormal decomposition (1675) (conferE.3.4) is a special case of biorthogonaldecomposition (1652) characterized by (1678). So, these characteristics (1654) are notnecessary conditions for biorthogonality.
Change language