v2009.01.01 - Convex Optimization

D.1. DIRECTIONAL DERIVATIVE, TAYLOR SERIES 619
in magnitude and direction to Y . D.3 Hence the directional derivative,
⎡
⎤
dg 11 (X) dg 12 (X) · · · dg 1N (X)
→Y
dg(X) =
∆ dg 21 (X) dg 22 (X) · · · dg 2N (X)
⎢
⎥
∈ R M×N
⎣ . . . ⎦
∣
dg M1 (X) dg M2 (X) · · · dg MN (X)
⎡
= ⎢
⎣
⎡
∣
dX→Y
tr ( ∇g 11 (X) T Y ) tr ( ∇g 12 (X) T Y ) · · · tr ( ∇g 1N (X) T Y ) ⎤
tr ( ∇g 21 (X) T Y ) tr ( ∇g 22 (X) T Y ) · · · tr ( ∇g 2N (X) T Y )
⎥
.
.
.
tr ( ∇g M1 (X) T Y ) tr ( ∇g M2 (X) T Y ) · · · tr ( ∇g MN (X) T Y ) ⎦
∑
k,l
∑
=
k,l
⎢
⎣ ∑
k,l
∂g 11 (X)
∂X kl
Y kl
∑
k,l
∂g 21 (X)
∂X kl
Y kl
∑
k,l
.
∂g M1 (X)
∑
∂X kl
Y kl
k,l
∂g 12 (X)
∂X kl
Y kl · · ·
∂g 22 (X)
∂X kl
Y kl · · ·
.
∂g M2 (X)
∂X kl
Y kl · · ·
∑
k,l
∑
k,l
∑
k,l
∂g 1N (X)
∂X kl
Y kl
∂g 2N (X)
∂X kl
Y kl
.
∂g MN (X)
∂X kl
⎤
⎥
⎦
Y kl
(1683)
from which it follows
→Y
dg(X) = ∑ k,l
∂g(X)
∂X kl
Y kl (1684)
Yet for all X ∈ domg , any Y ∈R K×L , and some open interval of t∈ R
g(X+ t Y ) = g(X) + t →Y
dg(X) + o(t 2 ) (1685)
which is the first-order Taylor series expansion about X . [190,18.4]
[128,2.3.4] Differentiation with respect to t and subsequent t-zeroing
isolates the second term of expansion. Thus differentiating and zeroing
g(X+ t Y ) in t is an operation equivalent to individually differentiating and
zeroing every entry g mn (X+ t Y ) as in (1682). So the directional derivative
of g(X) : R K×L →R M×N in any direction Y ∈ R K×L evaluated at X ∈ domg
becomes
→Y
dg(X) = d dt∣ g(X+ t Y ) ∈ R M×N (1686)
t=0
D.3 Although Y is a matrix, we may regard it as a vector in R KL .