MACHINE LEARNING TECHNIQUES - LASA

61
Figure 3-18: Linear combination of very simple classifiers (learners) trained by boosting. Each
classifier on its own would not be able to separate the data. Their linear combination is able to
separate complex data.[DEMOS\CLASSIFICATION\BOOSTING.ML]
Note:
Boosting and Bagging are methods based on a supervised process, where the “real” class of the
data is known a priori. In contrast, the clustering techniques seen in the first part of this chapter
work in an unsupervised manner, where the true labeling of the data is unknown.
3.3 Bayes Classifier
The simplest means to perform binary classification in probabilistic models is to use the so-called
i
y ∈− 1, + 1 of a set of i = 1.... M datapoints x i .
Bayes Classifier. Assume a binary labeling [ ]
Assume that you have built two probabilistic models p ( y| x) and p ( y|
x)
+ −
that predict the
probability that the data point x had associated label +1 and -1 respectively. A Bayes classifier will
decide on the correct labeling simply by comparing the relative probabilities of each model, i.e.:
( ) ≥ ( )
If p y| x p y| x , then y=+1.
+ −
Otherwise y=-1.
(3.32)
Clearly such a simplistic model is bound to be very erroneous as it does not take into account the
absolute value of the likelihood associated with each model. If both classifiers are predicting the
labeling of x with a very very low likelihood (which happens when the data point x is very far from
the training points or when the two classes overlap heavily in that region) then, deciding on one
class label over the other is usually no better than random. Besides, when p+ and p are two
−
arbitrary densities, comparing these may be dangerous if one did not make sure that the
© A.G.Billard 2004 – Last Update March 2011