v2009.01.01 - Convex Optimization

2.2. VECTORIZED-MATRIX INNER PRODUCT 45
2.2 Vectorized-matrix inner product
Euclidean space R n comes equipped with a linear vector inner-product
〈y,z〉 ∆ = y T z (26)
We prefer those angle brackets to connote a geometric rather than algebraic
perspective; e.g., vector y might represent a hyperplane normal (2.4.2).
Two vectors are orthogonal (perpendicular) to one another if and only if
their inner product vanishes;
y ⊥ z ⇔ 〈y,z〉 = 0 (27)
When orthogonal vectors each have unit norm, then they are orthonormal.
A vector inner-product defines Euclidean norm (vector 2-norm)
‖y‖ 2 = ‖y‖ ∆ = √ y T y , ‖y‖ = 0 ⇔ y = 0 (28)
For linear operation A on a vector, represented by a real matrix, the adjoint
operation A T is transposition and defined for matrix A by [197,3.10]
〈y,A T z〉 ∆ = 〈Ay,z〉 (29)
The vector inner-product for matrices is calculated just as it is for vectors;
by first transforming a matrix in R p×k to a vector in R pk by concatenating
its columns in the natural order. For lack of a better term, we shall call
that linear bijective (one-to-one and onto [197, App.A1.2]) transformation
vectorization. For example, the vectorization of Y = [y 1 y 2 · · · y k ] ∈ R p×k
[140] [284] is
⎡ ⎤
y 1
vec Y =
∆ y
⎢ 2
⎥
⎣ . ⎦ ∈ Rpk (30)
y k
Then the vectorized-matrix inner-product is trace of matrix inner-product;
for Z ∈ R p×k , [53,2.6.1] [173,0.3.1] [334,8] [318,2.2]
where (A.1.1)
〈Y , Z〉 ∆ = tr(Y T Z) = vec(Y ) T vec Z (31)
tr(Y T Z) = tr(ZY T ) = tr(YZ T ) = tr(Z T Y ) = 1 T (Y ◦ Z)1 (32)