v2009.01.01 - Convex Optimization

618 APPENDIX D. MATRIX CALCULUS
By simply rotating our perspective of the four-dimensional representation of
gradient matrix, we find one of three useful transpositions of this quartix
(connoted T 1 ):
⎡
∇g(X) T 1
=
⎢
⎣
∂g(X)
∂X 11
∂g(X)
∂X 21
.
∂g(X)
∂X K1
∂g(X)
∂X 12
· · ·
∂g(X)
∂X 22
.
· · ·
∂g(X)
∂X K2
· · ·
∂g(X)
∂X 1L
∂g(X)
∂X 2L
.
∂g(X)
∂X KL
⎤
⎥
⎦ ∈ RK×L×M×N (1677)
When the limit for ∆t∈ R exists, it is easy to show by substitution of
variables in (1674)
∂g mn (X) g mn (X + ∆t Y kl e
Y kl = lim
k e T l ) − g mn(X)
∂X kl ∆t→0 ∆t
∈ R (1678)
which may be interpreted as the change in g mn at X when the change in X kl
is equal to Y kl , the kl th entry of any Y ∈ R K×L . Because the total change
in g mn (X) due to Y is the sum of change with respect to each and every
X kl , the mn th entry of the directional derivative is the corresponding total
differential [190,15.8]
dg mn (X)| dX→Y
= ∑ k,l
∂g mn (X)
∂X kl
Y kl = tr ( ∇g mn (X) T Y ) (1679)
= ∑ g mn (X + ∆t Y kl e
lim
k e T l ) − g mn(X)
(1680)
∆t→0 ∆t
k,l
g mn (X + ∆t Y ) − g mn (X)
= lim
(1681)
∆t→0 ∆t
= d dt∣ g mn (X+ t Y ) (1682)
t=0
where t∈ R . Assuming finite Y , equation (1681) is called the Gâteaux
differential [36, App.A.5] [173,D.2.1] [311,5.28] whose existence is implied
by existence of the Fréchet differential (the sum in (1679)). [215,7.2] Each
may be understood as the change in g mn at X when the change in X is equal