MACHINE LEARNING TECHNIQUES - LASA

76
5.3 Kernel PCA
We start our review of kernel methods with kernel PCA, a non-linear extension to Principal
Component Analysis (PCA).
To recall, PCA is a powerful technique for extracting structure from high-dimensional data. In
Section 2.1 and Section 6.6.3, of these lecture notes, we review two ways of computing PCA. We
can either do it through an analytical decomposition into eigenvalues and eigenvectors of the data
space, or through an iterative decomposition using Hebbian Learning in multiple input-output
networks.
A key assumption of PCA was that the transformation applied on the data was linear. Kernel PCA
is a generalization of the standard PCA, in that it considers any linear or non-linear transformation
of the data. We first start by reintroducing the PCA notation and show how we can go from linear
PCA to non-linear PCA.
Linear PCA:
Assume that the dataset is composed of a set of vectors x
i
∈ °
N
, i = 1... M . Assume further that the
M
dataset is zero mean, i.e. ∑ x i
= 0.
i = 1
PCA finds an orthonormal basis such that the projections along each axis maximize the variance
of the data. To do so, PCA proceeds by diagonalizing the covariance matrix of the dataset C=XX
T
.
C is positive definite and can thus be diagonalized with non-negative eigenvalues.
The principal components are then all the vectors v
i
, i=
1..., N, solution of:
Cv
i
= λ
i
iv
(5.4)
where λ is a scalar and corresponds to the eigenvalue associated to the eigenvector v
i
i
.
Non-Linear Case:
Observe first that one can rewrite the eigenvalue decomposition problem of linear PCA in terms
i
of dot product across pairs of the training datapoints. Indeed, each principal component v can be
M
1 j j
T
expressed as a linear combination of the datapoints. Using C= ∑ x ( x ) and replacing in
M j = 1
(5.4), we obtain:
M
1
T
Cv = x ( x ) v = λiv
M
i j j i i
j=
1
which allows us to express each eigenvector as follows:
∑ (5.5)
© A.G.Billard 2004 – Last Update March 2011