v2009.01.01 - Convex Optimization

612 APPENDIX D. MATRIX CALCULUS
Gradient of vector-valued function g(X) : R K×L →R N on matrix domain
is a cubix
[
∇X(:,1) g 1 (X) ∇ X(:,1) g 2 (X) · · · ∇ X(:,1) g N (X)
∇g(X) ∆ =
∇ X(:,2) g 1 (X) ∇ X(:,2) g 2 (X) · · · ∇ X(:,2) g N (X)
...
...
...
∇ X(:,L) g 1 (X) ∇ X(:,L) g 2 (X) · · · ∇ X(:,L) g N (X) ]
= [ ∇g 1 (X) ∇g 2 (X) · · · ∇g N (X) ] ∈ R K×N×L (1645)
while the second-order gradient has a five-dimensional representation;
[
∇∇X(:,1) g 1 (X) ∇∇ X(:,1) g 2 (X) · · · ∇∇ X(:,1) g N (X)
∇ 2 g(X) ∆ =
∇∇ X(:,2) g 1 (X) ∇∇ X(:,2) g 2 (X) · · · ∇∇ X(:,2) g N (X)
...
...
...
∇∇ X(:,L) g 1 (X) ∇∇ X(:,L) g 2 (X) · · · ∇∇ X(:,L) g N (X) ]
= [ ∇ 2 g 1 (X) ∇ 2 g 2 (X) · · · ∇ 2 g N (X) ] ∈ R K×N×L×K×L (1646)
The gradient of matrix-valued function g(X) : R K×L →R M×N on matrix
domain has a four-dimensional representation called quartix
⎡
∇g(X) =
∆ ⎢
⎣
∇g 11 (X) ∇g 12 (X) · · · ∇g 1N (X)
∇g 21 (X) ∇g 22 (X) · · · ∇g 2N (X)
. .
∇g M1 (X) ∇g M2 (X) · · ·
.
∇g MN (X)
⎤
⎥
⎦ ∈ RM×N×K×L (1647)
while the second-order gradient has six-dimensional representation
⎡
∇ 2 g(X) =
∆ ⎢
⎣
and so on.
∇ 2 g 11 (X) ∇ 2 g 12 (X) · · · ∇ 2 g 1N (X)
∇ 2 g 21 (X) ∇ 2 g 22 (X) · · · ∇ 2 g 2N (X)
. .
∇ 2 g M1 (X) ∇ 2 g M2 (X) · · ·
.
∇ 2 g MN (X)
⎤
⎥
⎦ ∈ RM×N×K×L×K×L
(1648)