MACHINE LEARNING TECHNIQUES - LASA

22
It is clear that PCA can be used as the basis of a clustering method, by grouping the points
according to how they clump along each projection. Each cluster can then be treated separately.
Figure 2-1 illustrates a situation in which PCA is useful in projecting the dataset onto a new frame
of reference, from which the cluster of datapoints can be easily inferred. Figure 2-4 on the other
hand shows an example in which PCA would fail to segment two clusters. By projecting the data
along the principal direction, i.e. the vertical axis, PCA would merge the two clusters. ICA is one
alternative method for separating such clusters.
Figure 2-4: A classical illustration of problems occurring with variance-based methods such as PCA.
2.1.4 Projection Pursuit
Projection–Pursuit (PP) methods aim at finding structures in multivariate data by projecting them
on a lower-dimensional subspace. Since there are infinitely many projections from a higher
dimension to a lower dimension, PP aims at finding a technique to pursue a finite sequence of
projections that can reveal the most “interesting” structures of the data. The interesting
projections are often those that depart from that of normal/Gaussian distributions. For instance,
these structures can take the form of trends, clusters, hypersurfaces, or anomalies. Figure 2-4
illustrates the ``interestingness'' of non-gaussian projections. The data in this figure is clearly
divided into two clusters. However, the principal component, i.e. the direction of maximum
variance, would be vertical, providing no separation between the clusters. In contrast, the strongly
non-gaussian projection pursuit direction would be horizontal, providing optimal separation of the
clusters. Independent Component Analysis (see Section 2.3) is a method for discovering a linear
decomposition into directions that maximize non-Gaussianity. In this respect, it provides a first
step for PP decomposition.
Typically, PP uses a projection index, a functional computed on a projected density (or data set),
to measure the “interestingness” of the current projection and then uses a numerical optimizer to
move the projection direction to a more interesting position. Interesting projections usually refer to
non-linear structures in the data, i.e. structure than cannot be extracted by linear projections such
as PCA and ICA. PP can be formalized as follows:
Given a dataset
P
P
P P
X ∈° and a unit vector a ∈ ° one defines an index Ia
: ° → ° that
measures the interest of the associated projection P ( )
maximize I a
.
a
X . PP finds the parameters a that
© A.G.Billard 2004 – Last Update March 2011