MACHINE LEARNING TECHNIQUES - LASA

79
1 M
The solutions to the dual eigenvalue problem are given by all the eigenvectors α ,..., α with
non-zero eigenvalues λ ,..., λ .
1
M
i i
One Asking can that now the compute eigenvectors the projections v of Cφ
be of normalized, a given query i.e. point v , x v onto = 1 the ∀ ieigenvectors
= 1,. .., M
1 M
i
is equivalent to asking that the dual eigenvectors α ,..., α be such that 1/ λi
= α .
M
M
( ) = ∑ j ( ) ( ) = ∑ j ( )
(5.14)
i i j i j
v , φ x α φ x , φ x α k x , x
j= 1 j=
1
i
v
using:
Note that the solution of kernel PCA yields M eigenvectors, whereas the solution to linear PCA
yielded N eigenvectors, where N is the dimensionality of the dataset and M is the number of
datapoints. Usually M is much larger than N , hence kernel PCA corresponds to a lifting into a
higher-dimensional space, whereas linear PCA was a projection into a space of lower dimension
than the original space. By lifting the data, kernel PCA aims at extracting features that are
common to subsets of datapoints. In some way, this is close to a clustering technique.
Datapoints that bear some similarity will be close to one another along some particular projection.
If the data points have really no regularity, then they will be distributed homogeneously along all
projections.
Figure 5-2 illustrates the principle of PCA on a dataset that forms approximatively three clusters.
By looking at the regions with equal projection value on the first or second eigenvector, we see
that some subgroups of datapoints tend to group in the same region. When the kernel width is
large, the two clusters on the right-handside of the figure are encapsulated onto a single contour
line in the first projection. The datapoints of the cluster on the far left are closely grouped
together. As a result, the contour lines form ellipsoids that match the dispersion of the data. When
the datapoints are more loosely grouped, as it is the case for the two groups at the center and far
right, one observes some non-linear deformations of the contour lines that reflect the deflections
due to the absence of the datapoints. Using a smaller kernel width allows for encapsulating finer
features and allows us to separate the groups. A very small kernel width will however lead to one
datapoint per projection.
© A.G.Billard 2004 – Last Update March 2011