MACHINE LEARNING TECHNIQUES - LASA

59
C:
X → Y
( )
C x
K
= arg max ∑ 1
y k
kC : x = y
( )
(3.26)
The hope is, thus, that the aggregation of all the classifiers will give better classification results
than training a single classifier on the whole dataset.
If the classification method bases its estimation on the average of the data, for a very
heterogeneous dataset with local structures in the data, such an approach may fail to properly
represent the local structure unless the classifier is provided with enough granularity to
encapsulate these non-linearities 3 . A set of classifiers based on smaller subsets of the data may
have more chances to catch these local structures. However, training classifiers on only a small
subset of the data at hand may also degrade performance as they may fail to extract the generic
sets of features of the data at hand (because they see too small a set) and focus on noise or
particularities of each subset. This is a usual problem in machine learning and in classification, in
particular. One must find a tradeoff between generalizing (hence having much less parameters
than original datapoints) and representing all the local structures in the data.
3.2.3.2 Boosting / Adaboost
While bagging creates in parallel a set of K classifiers, boosting creates the classifiers
sequentially and uses each previously created classifier to boost training of the next classifier.
Principle:
A weight is associated to each datapoint of the training set as well as to each classifier. Weights
associated to the datapoint are adapted at each iteration step to reflect how well the datapoint is
predicted by the global classifier. The less well classified the data point, the bigger its associated
weight. This way, poorly classified data will be given more influence on the measure of the error
and will be more likely to be selected to train the new classifier created at each iteration. As a
result, they should be better estimated by this new classifier.
Similarly the classifiers are weighted when combined to form the final classifier, so as to reflect
their classification power. The poorer the classification power of a given classifier on the training
set associated with this classifier, the less influence the classifier is given for final classification.
Algorithm:
M
i i i N i
Let us consider a binary classification problem with X = { x , y } , x ∈ , y ∈ { + 1; −1}
v, i 1... M
i
i=
1
° . Let
= be the weights associated to each datapoint. Usually these are uniformly distributed
k
for starters and hence one builds a first set of l classifiers C , k 1,... l
all the set of data points.
= by drawing uniformly from
The final classifier is composed of a linear combination of the classifiers, so that each data point x
is then classified according to the function:
3 For instance in the case of multi-layer perceptron, one can add several hidden neurons and achieve optimal
performance. However in this case one may argue that each neuron in the hidden layer is some sort of subclassifier
(especially when using the threshold function as activation function for the output of the hidden neurons). Similarly in the
case of Support Vector Machines, one may increase the number of support vector points until reaching an optimal
description of all local non-linearities. At maximum, one may take all points as support vectors….!
© A.G.Billard 2004 – Last Update March 2011