MACHINE LEARNING TECHNIQUES - LASA

16
2 Methods for Correlation Analysis PCA, CCA, ICA
Principal Component Analysis (PCA), Canonical Correlation Analysis (CCA) and Independent
Component Analysis (ICA) are techniques for
• Discovering or reducing the dimensionality of a multidimensional data set
• Identifying a suitable representation of the multivariate data, by de-correlating the
dataset.
The principle of these techniques consists in determining the directions of the multidimensional
space of the dataset, along which the variability of the data is maximal. These directions
correspond to the principal axes or Eigenvectors of the correlation matrix applied to the whole
dataset. By projecting the data onto the referential defined by the Eigenvectors, one obtains a
representation of the data that minimizes the statistical dependence across the data.
Dimensionality reduction is obtained by discarding the dimensions along which the variance
appears to be smaller than a criterion; the data appear to be quasi constant along these
dimensions.
Identifying the best suitable representation of a dataset is fundamental as it can simplify
enormously the search for a solution. Consider the example of Figure 2-1 left. It is clear to a
human viewer that the data align themselves along an ellipse. However, an algorithm would have
trouble finding the regularities underlying the dataset, if the data coordinates are given with
respect to an external coordinate frame. On the other hand, the task becomes much simpler, if
the data are transferred to a coordinate frame alongside the axes of the ellipse, as illustrated in
Figure 2-1 right.
Figure 2-1: The two Eigenvectors determine the axes of an ellipse. The Eigenvalues determine the length of
the axes of an ellipse that fits best the data.
© A.G.Billard 2004 – Last Update March 2011