MACHINE LEARNING TECHNIQUES - LASA

60
with w ,..., 1 K
K
k
( ) ( )
C x = ∑ wkC x
k = 1
w the weights associated with each classifier.
Boosting builds classifiers incrementally so as to minimize classification error. To do this, one first
updates the weights associated to each datapoint. These weights represent the probability with
which the datapoint should be used. The less well the point is classified, the more likely it is to be
included in the training set for the new classifier.
Fist one computes how well each classifier represents the whole dataset by computing a local
error:
e
k
1
= ∑ vi
M
(3.27)
i k i i
x : C ( x ) ≠ y
The weight of each classifier is then computed to reflect how well this particular classifier
w = log 1/ β .
represents all the data at hand according to ( )
k
Then, one re-estimates the weight of each datapoint so as to give more weight to points that are
ek
misclassified. If β
k
= , k = 1... K
1−
e
all the weights of the datapoints.
k
, then
∑
vi
→ vi⋅β
k
k i i
C ( x ) = y
k
, followed by a normalization of
More recent advances in boosting techniques determine the weights associated to each classifier
by making a line search on the global error function. The error on the complete classifier is given
by:
i
( )
M
i
( ) sgn 1 ( )
∑ (3.28)
e x = −C x ⋅y
i=
1
Since this function is discontinuous and may change drastically as an effect of adding a new
classifier, one prefers to optimize an upper bound on this function with the following loss function:
i
( ( ))
M
i
( ( )) exp ( )
∑ (3.29)
LC X = − Cx ⋅y
i=
1
which is continuous and onto which one can then perform a line search on a sort of gradient
descent on the error. The local loss reduction resulting from adding a new classifier
associated weight w
k +
can be computed as follows::
1
k +
( ( ) 1
k + 1 ( ))
∂ LC X + w C X
∂w
k 1
C +
with
(3.30)
Deriving and recomputing the weights for each datapoint for the new classifier
with
v
i
=
M
e
∑
j=
1
k + 1
e
k +
i
( x )
j
1 ( x )
, one gets the same reformulation of the weight update as
w
1 ⎛⎛1−
e log
⎝⎝
k+
1
k+
1
= ⎜⎜ ⎟⎟
2 ek
+ 1
⎞⎞
⎠⎠
(3.31)
© A.G.Billard 2004 – Last Update March 2011