MACHINE LEARNING TECHNIQUES - LASA

20
Figure 2-2: An example of dimensionality reduction using PCA. The source images (left) are 32x32 color
pixels. Each image corresponds to a 3072 dimensional vector. (center) the principal components shown in
decreasing order of eigenvalue, notice how the first components contain the main features of the data (e.g.
color of the balls) while the components further down only contain fine details. (right) projection of the source
images onto the first two principal components.
We are here faced with contradictory goals: On one hand, we should simplify the problem by
reducing the dimension of the representation. On the other hand we want to preserve as much as
possible of the original information content. PCA offers a convenient way to control the trade-off
between loosing information and simplifying the problem at hand.
As it will be noted later, it may be possible to create piecewise linear models by dividing the input
data to smaller regions and fitting linear models locally to the data.
2.1.2 Solving PCA as an optimization under constraint problem
The standard PCA procedure delineated in the previous paragraphs can also be rephrased as an
optimization under constraint problem in two different ways.
2.1.2.1 Variance Maximization through constrained optimization
Observe first that by projecting the original dataset onto the eigenvectors, PCA requires that the
projection form an orthonormal basis. In addition, by projecting onto the eigenvector of the
covariance matrix, it ensures that the first projection is along the direction of maximal variance of
the datat.
This can be formulated as an optimization problem that maximizes the following objective
function:
with
C
M
1
1
T
T
arg max J ( e ,..., e ) = arg max ( e ) x = ( e ) Ce
j j∈ {1,... N} M i=
1
the covariance of the dataset (which is first made zero-mean).
∑ (2.7)
N j i j j
Adding the constraint that the e
( )
j k j
e = 1, ∀ j = 1... q and e e = 0 ∀k ≠ j.
T
j
should for an orthonormal basis
PCA becomes an optimization under constraint problem which can be solved using Lagrange
multipliers. PCA proceeds iteratively by first solving for the first eigenvector, using:
T
( )
T
( ) ( ) λ 1 1 ( )
1 1 1 1 1
Le = e Ce− − e e
(2.8)
where 1
λ is the first Lagrange multiplier, and then solving iteratively for all other eigenvectors,
adding the orthogonality constraint.
© A.G.Billard 2004 – Last Update March 2011