MACHINE LEARNING TECHNIQUES - LASA

21
2.1.2.2 Reconstruction error minimization through constrained optimization
Earlier on, we showed that PCA finds the optimal (in a mean-square sense) projections of the
dataset. This, again, can be formalized as an optimization under constraint problem taking, that
minimizes the following objective function:
T i
( ex
j )
1
( ,..., , )
q
M
q
1
i
J e e λ = x −µ − λje
M
∑ ∑
j
(2.9)
i= 1 j=
1
where λ = are the projection coefficients and xthe mean of the data.
ij
One optimizes J under the constraints that the eigenvectors form an orthonormal basis, i.e:
j
( )
T
i
i
e = 1 and e ⋅ e = 0, ∀ i, j = 1,..., q.
2.1.3 PCA limitations
PCA is a simple, straightforward means of determining the major dimensions of a dataset. It
suffers, however, from a number of drawbacks. The principal components found by projecting the
dataset onto the perpendicular basis vectors (eigenvectors) are uncorrelated, and their directions
orthogonal. The assumption that the referential is orthogonal is often too constraining, see Figure
2-3 for an illustration.
Figure 2-3: Assume a set of data points whose joint distribution forms a parallelogram. The first PC is the
direction with the greatest spread, along the longest axis of the parallelogram. The second PC is orthogonal
to the first one, by necessity. The independent component directions are, however, parallel to the sides of
the parallelogram.
PCA ensures only uncorrelatedness. This is a less constraining condition than statistical
independence, which makes standard PCA ill suited for dealing with non-Gaussian data. ICA is a
method that specifically ensures statistical independence.
© A.G.Billard 2004 – Last Update March 2011