v2010.10.26 - Convex Optimization
v2010.10.26 - Convex Optimization v2010.10.26 - Convex Optimization
686 APPENDIX D. MATRIX CALCULUSIn the case g(X) : R K →R has vector argument, they further simplify:and so on.D.1.7Taylor series→Ydg(X) = ∇g(X) T Y (1834)→Ydg 2 (X) = Y T ∇ 2 g(X)Y (1835)→Ydg 3 (X) = ∇ X(Y T ∇ 2 g(X)Y ) TY (1836)Series expansions of the differentiable matrix-valued function g(X) , ofmatrix argument, were given earlier in (1808) and (1829). Assuming g(X)has continuous first-, second-, and third-order gradients over the open setdomg , then for X ∈ domg and any Y ∈ R K×L the complete Taylor seriesis expressed on some open interval of µ∈Rg(X+µY ) = g(X) + µ dg(X) →Y+ 1 →Yµ2dg 2 (X) + 1 →Yµ3dg 3 (X) + o(µ 4 ) (1837)2! 3!or on some open interval of ‖Y ‖ 2g(Y ) = g(X) +→Y −X→Y −X→Y −Xdg(X) + 1 dg 2 (X) + 1 dg 3 (X) + o(‖Y ‖ 4 ) (1838)2! 3!which are third-order expansions about X . The mean value theorem fromcalculus is what insures finite order of the series. [219] [42,1.1] [41, App.A.5][199,0.4] These somewhat unbelievable formulae imply that a function canbe determined over the whole of its domain by knowing its value and all itsdirectional derivatives at a single point X .D.1.7.0.1 Example. Inverse-matrix function.Say g(Y )= Y −1 . From the table on page 692,→Ydg(X) = d dt∣ g(X+ t Y ) = −X −1 Y X −1 (1839)t=0→Ydg 2 (X) = d2dt 2 ∣∣∣∣t=0g(X+ t Y ) = 2X −1 Y X −1 Y X −1 (1840)→Ydg 3 (X) = d3dt 3 ∣∣∣∣t=0g(X+ t Y ) = −6X −1 Y X −1 Y X −1 Y X −1 (1841)
D.1. DIRECTIONAL DERIVATIVE, TAYLOR SERIES 687Let’s find the Taylor series expansion of g about X = I : Since g(I)= I , for‖Y ‖ 2 < 1 (µ = 1 in (1837))g(I + Y ) = (I + Y ) −1 = I − Y + Y 2 − Y 3 + ... (1842)If Y is small, (I + Y ) −1 ≈ I − Y . D.3 Now we find Taylor series expansionabout X :g(X + Y ) = (X + Y ) −1 = X −1 − X −1 Y X −1 + 2X −1 Y X −1 Y X −1 − ...(1843)If Y is small, (X + Y ) −1 ≈ X −1 − X −1 Y X −1 .D.1.7.0.2 Exercise. log det. (confer [61, p.644])Find the first three terms of a Taylor series expansion for log detY . Specifyan open interval over which the expansion holds in vicinity of X . D.1.8Correspondence of gradient to derivativeFrom the foregoing expressions for directional derivative, we derive arelationship between the gradient with respect to matrix X and the derivativewith respect to real variable t :D.1.8.1first-orderRemoving evaluation at t = 0 from (1809), D.4 we find an expression for thedirectional derivative of g(X) in direction Y evaluated anywhere along aline {X+ t Y | t ∈ R} intersecting domg→Ydg(X+ t Y ) = d g(X+ t Y ) (1844)dtIn the general case g(X) : R K×L →R M×N , from (1802) and (1805) we findtr ( ∇ X g mn (X+ t Y ) T Y ) = d dt g mn(X+ t Y ) (1845)D.3 Had we instead set g(Y )=(I + Y ) −1 , then the equivalent expansion would have beenabout X =0.D.4 Justified by replacing X with X+ tY in (1802)-(1804); beginning,dg mn (X+ tY )| dX→Y= ∑ k, l∂g mn (X+ tY )Y kl∂X kl
- Page 635 and 636: B.1. RANK-ONE MATRIX (DYAD) 635R(v)
- Page 637 and 638: B.1. RANK-ONE MATRIX (DYAD) 637B.1.
- Page 639 and 640: B.2. DOUBLET 639R([u v ])R(Π)= R([
- Page 641 and 642: B.3. ELEMENTARY MATRIX 641has N −
- Page 643 and 644: B.4. AUXILIARY V -MATRICES 643is an
- Page 645 and 646: B.4. AUXILIARY V -MATRICES 64514. [
- Page 647 and 648: B.5. ORTHOGONAL MATRIX 647Given X
- Page 649 and 650: B.5. ORTHOGONAL MATRIX 649Figure 15
- Page 651 and 652: B.5. ORTHOGONAL MATRIX 651which is
- Page 653 and 654: Appendix CSome analytical optimal r
- Page 655 and 656: C.2. TRACE, SINGULAR AND EIGEN VALU
- Page 657 and 658: C.2. TRACE, SINGULAR AND EIGEN VALU
- Page 659 and 660: C.2. TRACE, SINGULAR AND EIGEN VALU
- Page 661 and 662: C.3. ORTHOGONAL PROCRUSTES PROBLEM
- Page 663 and 664: C.4. TWO-SIDED ORTHOGONAL PROCRUSTE
- Page 665 and 666: C.4. TWO-SIDED ORTHOGONAL PROCRUSTE
- Page 667 and 668: C.4. TWO-SIDED ORTHOGONAL PROCRUSTE
- Page 669 and 670: Appendix DMatrix calculusFrom too m
- Page 671 and 672: D.1. DIRECTIONAL DERIVATIVE, TAYLOR
- Page 673 and 674: D.1. DIRECTIONAL DERIVATIVE, TAYLOR
- Page 675 and 676: D.1. DIRECTIONAL DERIVATIVE, TAYLOR
- Page 677 and 678: D.1. DIRECTIONAL DERIVATIVE, TAYLOR
- Page 679 and 680: D.1. DIRECTIONAL DERIVATIVE, TAYLOR
- Page 681 and 682: D.1. DIRECTIONAL DERIVATIVE, TAYLOR
- Page 683 and 684: D.1. DIRECTIONAL DERIVATIVE, TAYLOR
- Page 685: D.1. DIRECTIONAL DERIVATIVE, TAYLOR
- Page 689 and 690: D.1. DIRECTIONAL DERIVATIVE, TAYLOR
- Page 691 and 692: D.2. TABLES OF GRADIENTS AND DERIVA
- Page 693 and 694: D.2. TABLES OF GRADIENTS AND DERIVA
- Page 695 and 696: D.2. TABLES OF GRADIENTS AND DERIVA
- Page 697 and 698: D.2. TABLES OF GRADIENTS AND DERIVA
- Page 699 and 700: Appendix EProjectionFor any A∈ R
- Page 701 and 702: 701U T = U † for orthonormal (inc
- Page 703 and 704: E.1. IDEMPOTENT MATRICES 703where A
- Page 705 and 706: E.1. IDEMPOTENT MATRICES 705order,
- Page 707 and 708: E.1. IDEMPOTENT MATRICES 707are lin
- Page 709 and 710: E.3. SYMMETRIC IDEMPOTENT MATRICES
- Page 711 and 712: E.3. SYMMETRIC IDEMPOTENT MATRICES
- Page 713 and 714: E.3. SYMMETRIC IDEMPOTENT MATRICES
- Page 715 and 716: E.4. ALGEBRA OF PROJECTION ON AFFIN
- Page 717 and 718: E.5. PROJECTION EXAMPLES 717a ∗ 2
- Page 719 and 720: E.5. PROJECTION EXAMPLES 719where Y
- Page 721 and 722: E.5. PROJECTION EXAMPLES 721(B.4.2)
- Page 723 and 724: E.6. VECTORIZATION INTERPRETATION,
- Page 725 and 726: E.6. VECTORIZATION INTERPRETATION,
- Page 727 and 728: E.6. VECTORIZATION INTERPRETATION,
- Page 729 and 730: E.7. PROJECTION ON MATRIX SUBSPACES
- Page 731 and 732: E.7. PROJECTION ON MATRIX SUBSPACES
- Page 733 and 734: E.8. RANGE/ROWSPACE INTERPRETATION
- Page 735 and 736: E.9. PROJECTION ON CONVEX SET 735As
D.1. DIRECTIONAL DERIVATIVE, TAYLOR SERIES 687Let’s find the Taylor series expansion of g about X = I : Since g(I)= I , for‖Y ‖ 2 < 1 (µ = 1 in (1837))g(I + Y ) = (I + Y ) −1 = I − Y + Y 2 − Y 3 + ... (1842)If Y is small, (I + Y ) −1 ≈ I − Y . D.3 Now we find Taylor series expansionabout X :g(X + Y ) = (X + Y ) −1 = X −1 − X −1 Y X −1 + 2X −1 Y X −1 Y X −1 − ...(1843)If Y is small, (X + Y ) −1 ≈ X −1 − X −1 Y X −1 .D.1.7.0.2 Exercise. log det. (confer [61, p.644])Find the first three terms of a Taylor series expansion for log detY . Specifyan open interval over which the expansion holds in vicinity of X . D.1.8Correspondence of gradient to derivativeFrom the foregoing expressions for directional derivative, we derive arelationship between the gradient with respect to matrix X and the derivativewith respect to real variable t :D.1.8.1first-orderRemoving evaluation at t = 0 from (1809), D.4 we find an expression for thedirectional derivative of g(X) in direction Y evaluated anywhere along aline {X+ t Y | t ∈ R} intersecting domg→Ydg(X+ t Y ) = d g(X+ t Y ) (1844)dtIn the general case g(X) : R K×L →R M×N , from (1802) and (1805) we findtr ( ∇ X g mn (X+ t Y ) T Y ) = d dt g mn(X+ t Y ) (1845)D.3 Had we instead set g(Y )=(I + Y ) −1 , then the equivalent expansion would have beenabout X =0.D.4 Justified by replacing X with X+ tY in (1802)-(1804); beginning,dg mn (X+ tY )| dX→Y= ∑ k, l∂g mn (X+ tY )Y kl∂X kl
Change language