v2009.01.01 - Convex Optimization

D.1. DIRECTIONAL DERIVATIVE, TAYLOR SERIES 623
Rotating our perspective, we get several views of the second-order gradient:
⎡
∇ 2 g(X) = ⎢
⎣
∇ 2 g 11 (X) ∇ 2 g 12 (X) · · · ∇ 2 g 1N (X)
∇ 2 g 21 (X) ∇ 2 g 22 (X) · · · ∇ 2 g 2N (X)
. .
∇ 2 g M1 (X) ∇ 2 g M2 (X) · · ·
.
∇ 2 g MN (X)
⎤
⎥
⎦ ∈ RM×N×K×L×K×L (1695)
⎡
∇ 2 g(X) T 1
=
⎢
⎣
∇ ∂g(X)
∂X 11
∇ ∂g(X)
∂X 21
.
∇ ∂g(X)
∂X K1
∇ ∂g(X)
∂X 12
· · · ∇ ∂g(X)
∂X 1L
∇ ∂g(X)
∂X 22
.
· · · ∇ ∂g(X)
∂X 2L
.
∇ ∂g(X)
∂X K2
· · · ∇ ∂g(X)
∂X KL
⎤
⎥
⎦ ∈ RK×L×M×N×K×L (1696)
⎡
∇ 2 g(X) T 2
=
⎢
⎣
∂∇g(X)
∂X 11
∂∇g(X)
∂X 21
.
∂∇g(X)
∂X K1
∂∇g(X)
∂X 12
· · ·
∂∇g(X)
∂X 22
.
· · ·
∂∇g(X)
∂X K2
· · ·
∂∇g(X)
∂X 1L
∂∇g(X)
∂X 2L
.
∂∇g(X)
∂X KL
⎤
⎥
⎦ ∈ RK×L×K×L×M×N (1697)
Assuming the limits exist, we may state the partial derivative of the mn th
entry of g with respect to the kl th and ij th entries of X ;
∂ 2 g mn(X)
∂X kl ∂X ij
=
g mn(X+∆t e
lim
k e T l +∆τ e i eT j )−gmn(X+∆t e k eT l )−(gmn(X+∆τ e i eT)−gmn(X))
j
∆τ,∆t→0
∆τ ∆t
(1698)
Differentiating (1678) and then scaling by Y ij
∂ 2 g mn(X)
∂X kl ∂X ij
Y kl Y ij = lim
∆t→0
∂g mn(X+∆t Y kl e k e T l )−∂gmn(X)
∂X ij
Y
∆t
ij
(1699)
g mn(X+∆t Y kl e
= lim
k e T l +∆τ Y ij e i e T j )−gmn(X+∆t Y kl e k e T l )−(gmn(X+∆τ Y ij e i e T)−gmn(X))
j
∆τ,∆t→0
∆τ ∆t