v2010.10.26 - Convex Optimization
v2010.10.26 - Convex Optimization v2010.10.26 - Convex Optimization
250 CHAPTER 3. GEOMETRY OF CONVEX FUNCTIONS21.510.5Y 20−0.5−1−1.5−2−2 −1.5 −1 −0.5 0 0.5 1 1.5 2Figure 76: Gradient in R 2 evaluated on grid over some open disc in domainof convex quadratic bowl f(Y )= Y T Y : R 2 → R illustrated in Figure 77.Circular contours are level sets; each defined by a constant function-value.Y 13.6 GradientGradient ∇f of any differentiable multidimensional function f (formallydefined inD.1) maps each entry f i to a space having the same dimensionas the ambient space of its domain. Notation ∇f is shorthand for gradient∇ x f(x) of f with respect to x . ∇f(y) can mean ∇ y f(y) or gradient∇ x f(y) of f(x) with respect to x evaluated at y ; a distinction that shouldbecome clear from context.Gradient of a differentiable real function f(x) : R K →R with respect toits vector argument is uniquely defined⎡∇f(x) =⎢⎣∂f(x)∂x 1∂f(x)∂x 2.∂f(x)∂x K⎤⎥⎦ ∈ RK (1759)while the second-order gradient of the twice differentiable real function with
3.6. GRADIENT 251respect to its vector argument is traditionally called the Hessian ; 3.17⎡⎤∂ 2 f(x)∂x 1 ∂x 2· · ·∇ 2 f(x) =⎢⎣∂ 2 f(x)∂x 2 1∂ 2 f(x)∂x 2 ∂x 1.∂ 2 f(x)∂x K ∂x 1∂ 2 f(x)∂x 2 2· · ·....∂ 2 f(x)∂x K ∂x 2· · ·∂ 2 f(x)∂x 1 ∂x K∂ 2 f(x)∂x 2 ∂x K.∂ 2 f(x)∂x 2 K∈ S K (1760)⎥⎦The gradient can be interpreted as a vector pointing in the direction ofgreatest change, [219,15.6] or polar to direction of steepest descent. 3.18 [375]The gradient can also be interpreted as that vector normal to a sublevel set;e.g., Figure 78, Figure 67.For the quadratic bowl in Figure 77, the gradient maps to R 2 ; illustratedin Figure 76. For a one-dimensional function of real variable, for example,the gradient evaluated at any point in the function domain is just the slope(or derivative) of that function there. (conferD.1.4.1)For any differentiable multidimensional function, zero gradient ∇f = 0is a condition necessary for its unconstrained minimization [152,3.2]:3.6.0.0.1 Example. Projection on rank-1 subset.For A∈ S N having eigenvalues λ(A)= [λ i ]∈ R N , consider the unconstrainednonconvex optimization that is a projection on the rank-1 subset (2.9.2.1)of positive semidefinite cone S N + : Defining λ 1 max i {λ(A) i } andcorresponding eigenvector v 1minimizexarg minimizex‖xx T − A‖ 2 F = minimize tr(xx T (x T x) − 2Axx T + A T A)x{‖λ(A)‖ 2 , λ 1 ≤ 0=‖λ(A)‖ 2 − λ 2 (1695)1 , λ 1 > 0{0 , λ1 ≤ 0‖xx T − A‖ 2 F = √ (1696)v 1 λ1 , λ 1 > 0From (1789) andD.2.1, the gradient of ‖xx T − A‖ 2 F is∇ x((x T x) 2 − 2x T Ax ) = 4(x T x)x − 4Ax (590)3.17 Jacobian is the Hessian transpose, so commonly confused in matrix calculus.3.18 Newton’s direction −∇ 2 f(x) −1 ∇f(x) is better for optimization of nonlinear functionswell approximated locally by a quadratic function. [152, p.105]
- Page 199 and 200: 2.13. DUAL CONE & GENERALIZED INEQU
- Page 201 and 202: 2.13. DUAL CONE & GENERALIZED INEQU
- Page 203 and 204: 2.13. DUAL CONE & GENERALIZED INEQU
- Page 205 and 206: 2.13. DUAL CONE & GENERALIZED INEQU
- Page 207 and 208: 2.13. DUAL CONE & GENERALIZED INEQU
- Page 209 and 210: 2.13. DUAL CONE & GENERALIZED INEQU
- Page 211 and 212: 2.13. DUAL CONE & GENERALIZED INEQU
- Page 213 and 214: 2.13. DUAL CONE & GENERALIZED INEQU
- Page 215 and 216: 2.13. DUAL CONE & GENERALIZED INEQU
- Page 217 and 218: 2.13. DUAL CONE & GENERALIZED INEQU
- Page 219 and 220: Chapter 3Geometry of convex functio
- Page 221 and 222: 3.1. CONVEX FUNCTION 221f 1 (x)f 2
- Page 223 and 224: 3.1. CONVEX FUNCTION 223Rf(b)f(X
- Page 225 and 226: 3.2. PRACTICAL NORM FUNCTIONS, ABSO
- Page 227 and 228: 3.2. PRACTICAL NORM FUNCTIONS, ABSO
- Page 229 and 230: 3.2. PRACTICAL NORM FUNCTIONS, ABSO
- Page 231 and 232: 3.2. PRACTICAL NORM FUNCTIONS, ABSO
- Page 233 and 234: 3.2. PRACTICAL NORM FUNCTIONS, ABSO
- Page 235 and 236: 3.3. INVERTED FUNCTIONS AND ROOTS 2
- Page 237 and 238: 3.4. AFFINE FUNCTION 237rather]x >
- Page 239 and 240: 3.4. AFFINE FUNCTION 239f(z)Az 2z 1
- Page 241 and 242: 3.5. EPIGRAPH, SUBLEVEL SET 241{a T
- Page 243 and 244: 3.5. EPIGRAPH, SUBLEVEL SET 243Subl
- Page 245 and 246: 3.5. EPIGRAPH, SUBLEVEL SET 245wher
- Page 247 and 248: 3.5. EPIGRAPH, SUBLEVEL SET 247part
- Page 249: 3.5. EPIGRAPH, SUBLEVEL SET 249that
- Page 253 and 254: 3.6. GRADIENT 253Invertibility is g
- Page 255 and 256: 3.6. GRADIENT 2553.6.1.0.2 Theorem.
- Page 257 and 258: 3.6. GRADIENT 257f(Y )[ ∇f(X)−1
- Page 259 and 260: 3.6. GRADIENT 259αβα ≥ β ≥
- Page 261 and 262: 3.6. GRADIENT 2613.6.4 second-order
- Page 263 and 264: 3.7. CONVEX MATRIX-VALUED FUNCTION
- Page 265 and 266: 3.7. CONVEX MATRIX-VALUED FUNCTION
- Page 267 and 268: 3.7. CONVEX MATRIX-VALUED FUNCTION
- Page 269 and 270: 3.8. QUASICONVEX 269exponential alw
- Page 271 and 272: 3.9. SALIENT PROPERTIES 2713.8.0.0.
- Page 273 and 274: Chapter 4Semidefinite programmingPr
- Page 275 and 276: 4.1. CONIC PROBLEM 275(confer p.162
- Page 277 and 278: 4.1. CONIC PROBLEM 277PCsemidefinit
- Page 279 and 280: 4.1. CONIC PROBLEM 279is the affine
- Page 281 and 282: 4.1. CONIC PROBLEM 281faces of S 3
- Page 283 and 284: 4.1. CONIC PROBLEM 2834.1.2.3 Previ
- Page 285 and 286: 4.2. FRAMEWORK 285Semidefinite Fark
- Page 287 and 288: 4.2. FRAMEWORK 287On the other hand
- Page 289 and 290: 4.2. FRAMEWORK 2894.2.2.1 Dual prob
- Page 291 and 292: 4.2. FRAMEWORK 291For symmetric pos
- Page 293 and 294: 4.2. FRAMEWORK 293has norm ‖x ⋆
- Page 295 and 296: 4.2. FRAMEWORK 295minimize 1 TˆxX
- Page 297 and 298: 4.2. FRAMEWORK 297asminimize ‖ỹ
- Page 299 and 300: 4.3. RANK REDUCTION 2994.3 Rank red
250 CHAPTER 3. GEOMETRY OF CONVEX FUNCTIONS21.510.5Y 20−0.5−1−1.5−2−2 −1.5 −1 −0.5 0 0.5 1 1.5 2Figure 76: Gradient in R 2 evaluated on grid over some open disc in domainof convex quadratic bowl f(Y )= Y T Y : R 2 → R illustrated in Figure 77.Circular contours are level sets; each defined by a constant function-value.Y 13.6 GradientGradient ∇f of any differentiable multidimensional function f (formallydefined inD.1) maps each entry f i to a space having the same dimensionas the ambient space of its domain. Notation ∇f is shorthand for gradient∇ x f(x) of f with respect to x . ∇f(y) can mean ∇ y f(y) or gradient∇ x f(y) of f(x) with respect to x evaluated at y ; a distinction that shouldbecome clear from context.Gradient of a differentiable real function f(x) : R K →R with respect toits vector argument is uniquely defined⎡∇f(x) =⎢⎣∂f(x)∂x 1∂f(x)∂x 2.∂f(x)∂x K⎤⎥⎦ ∈ RK (1759)while the second-order gradient of the twice differentiable real function with