MACHINE LEARNING TECHNIQUES - LASA

17
2.1 Principal Component Analysis
Principal component analysis (PCA) performs a linear transformation of the coordinate system, so
as to maximize the variance of the data along the first principal axis of the new coordinate
system, e.g. the first axis of the ellipse as illustrated in Figure 2-1.
It is important that the data on which PCA is performed are correlated. If the data are
independent, nothing can be achieved with PCA.
Formalism:
Consider a data set of
i=
1...
{ j} j=
1,...
M
MN -dimensional data points
i i N
X= x and x ∈ ° , i = 1,..., M :
N
PCA aims at finding a linear map A , such that:
N A q
A: ° ⎯⎯⎯⎯→ ° , with q ≤ N
1
{ }
X ⎯⎯⎯⎯→ Y = AX , with Y = y ,...., y and each y ∈ °
A M i q
Algorithm:
Classical batch algorithm for PCA goes as follows:
The mean of the dataset is denoted by
⎛⎛x
+ .. + x
⎝⎝ M
+ .. + x
M
1 M 1
M
1 1 N N
= ( 1, 1,..., N ) = E( X)
=⎜⎜ ,..., ⎟⎟
µ µ µ µ
and the covariance matrix of the same data set is
1 1
⎛⎛x1 −µ 1,.........,
xN
−µ
⎞⎞
T
N
T BB ⎜⎜
⎟⎟
C = E{ X '( X ')
} = , B = ......................................
M ⎜⎜ ⎟⎟
⎜⎜ M
M
x1 µ
1,.........,
x1
µ ⎟⎟
⎝⎝ −
−
N ⎠⎠
whereby X ' = X- µ , i.e. X' is zero mean X.
x
⎞⎞
⎠⎠
(2.1)
(2.2)
The component cii
is the variance of the vector of components i of all data points, denoted X .
i
The variance of a component indicates the spread of the component values around its mean
value. The components of C, denoted by
1
c ( x )( x )
M
k
k
ij
= ∑ i
−µ i j
−µ
j
M k = 1
, are a measure of the
© A.G.Billard 2004 – Last Update March 2011