MACHINE LEARNING TECHNIQUES - LASA

58
When classification results from a linear projection through a map A, as in LDA, then again the
problem corresponds to maximizing J( A)
.
T
ASA
b
= with S ,
w
T
ASA
w
Sb
defined as above.
Figure 3-17: Elongated data from two classes: the black samples are spread more widely than
the white ones (left), the direction of maximum variance found by PCA is not optimal for
separating the two classes. LDA is able to find a better projection but it still does not allow linear
separation. Fisher Linear Discriminant takes into account the difference in variance of the two
distributions and finds a separable projection. [DEMOS\PROJECTION\PCA-LDA-FISHER.ML]
3.2.3 Mixture of linear classifiers (boosting and bagging)
Adapted from Bagging Predictors by Leo Breiman, Machine Learning, 24, 123–140 (1996)
Linear classifiers as described previously rely on regression and can perform only binary
classification. To extend the domain of application of classifiers to allow classification of more
than two classes, linear classifiers can be combined. A number of combining techniques can be
used, among which bagging and boosting have appeared in recent years as the most popular
methods due to their simplicity. All of these methods modify the training data set, build classifiers
on these modified training sets, and then combine them into a final decision rule by simple or
weighted majority voting. However, they perform in a different way.
We briefly review bagging and boosting next. Other multi-class classification methods which we
will see in this course are the multi-layer perceptron and multi-class support vector machines.
3.2.3.1 Bagging
Let us assume, once more, that the complete training set is composed of M N-dimensional data
M i N
points X { x} 1
, x ,
i
= ∈° with associated labels { } ,
1
i=
M
i
= ∈• . The labels are not
i=
Y y y
necessarily binary and hence this allows multi-class classification.
Bagging consists in selecting (drawing with replacement) at random K subsets of training
data
k
X k 1,...
K
{ }
k=
1
k k
= K . Each of these subsets will be used to create a classifier C ( X )
. Each
k
classifier performs a mapping C : X → Y. The final classifier is built from all classifiers and is
such that it outputs the class predicted most often by all K classifiers:
© A.G.Billard 2004 – Last Update March 2011