v2009.01.01 - Convex Optimization

222 CHAPTER 3. GEOMETRY OF CONVEX FUNCTIONS
respect to its vector argument is traditionally called the Hessian ; 3.12
⎡
∇ 2 f(x) =
∆ ⎢
⎣
∂ 2 f(x)
∂ 2 x 1
∂ 2 f(x)
∂x 2 ∂x 1
.
∂ 2 f(x)
∂x K ∂x 1
∂ 2 f(x)
∂x 1 ∂x 2
· · ·
∂ 2 f(x)
∂ 2 x 2
· · ·
.
...
∂ 2 f(x)
∂x K ∂x 2
· · ·
∂ 2 f(x)
∂x 1 ∂x K
∂ 2 f(x)
∂x 2 ∂x K
.
∂ 2 f(x)
∂ 2 x K
⎤
∈ S K (1638)
⎥
⎦
The gradient can be interpreted as a vector pointing in the direction of
greatest change. [190,15.6] The gradient can also be interpreted as that
vector normal to a level set; e.g., Figure 68, Figure 59.
For the quadratic bowl in Figure 67, the gradient maps to R 2 ; illustrated
in Figure 66. For a one-dimensional function of real variable, for example,
the gradient evaluated at any point in the function domain is just the slope
(or derivative) of that function there. (conferD.1.4.1)
For any differentiable multidimensional function, zero gradient ∇f = 0
is a necessary condition for its unconstrained minimization [128,3.2]:
3.1.8.0.1 Example. Projection on a rank-1 subset.
For A∈ S N having eigenvalues λ(A)= [λ i ]∈ R N , consider the unconstrained
nonconvex optimization that is a projection on the rank-1 subset (2.9.2.1)
of positive semidefinite cone S N + : Defining λ 1 ∆ = max
i
{λ(A) i } and
corresponding eigenvector v 1
minimize
x
‖xx T − A‖ 2 F = minimize tr(xx T (x T x) − 2Axx T + A T A)
x
{
‖λ(A)‖ 2 , λ 1 ≤ 0
=
‖λ(A)‖ 2 − λ 2 (1576)
1 , λ 1 > 0
arg minimize
x
‖xx T − A‖ 2 F =
{
0 , λ1 ≤ 0
v 1
√
λ1 , λ 1 > 0
(1577)
From (1666) andD.2.1, the gradient of ‖xx T − A‖ 2 F is
∇ x
(
(x T x) 2 − 2x T Ax ) = 4(x T x)x − 4Ax (527)
3.12 Jacobian is the Hessian transpose, so commonly confused in matrix calculus.