MACHINE LEARNING TECHNIQUES - LASA

62
quantities are comparable. Since a pdf is a positive, but unbounded (above), function, its
likelihood may grow arbitrarily. It usually grows with the numer of parameters. A good precaution
is then to make sure that the two classifiers have the same number of parameters and are trained
on the same number of datapoints. The latter means that one must have at its disposal as many
examples of the positive class as that of the negative class. If this can not be ensured, then one
may apply some normalization term on the likelihood of each classifier or one may use a
threshold on the likelihood of both models to determine when inference is warranted. Despite
these caveats, Bayes classifiers remain quite popular, in part because of their extreme simplicity,
but also because they yield very good performance in practice.
Note that Bayes classification can easily be extended to multiclass classification. Assume that
i
you wish to classify a set of datapoints { }
i=
1
M
X = x into K classes
i
i
y associated to each datapoint x , i = 1.... M,
can then take any value
compute the posterior probability of each class, using:
1 K
C ,...., C . Each label
1 K
C ,...., C . One can then
i
( | )
p y = C x =
i
i
p( y = C ) p( x|
y = C )
K
k
k
∑ p( y = C ) p( x|
y = C )
k = 1
(3.33)
We see next how this can be used when using Gaussian Mixture Models. Note that we will revisit
multi-class classification using kernel methods in Section 5.10.
3.4 Linear classification with Gaussian Mixture Models
The Gaussian Mixture Model, introduced in Section 3.1.5, can be used in conjunction with the
Bayes classifier to perform binary classification. For two knows classes of datapoints, one can
train two separate GMM-s, one for each class of datapoints and use the Bayes classifier to
determine the labeling of any given datapoint. In other word, each GMM yields a density
p and p which can then be used to determine the label of a new training point using (3.32).
+ −
To ensure that the two models are comparable, one should train two GMM with same number of
Gaussians, using the same modeling (with full, diagonal or isotropic covariance matrix in each
case), and using similar number of training points for each GMM. Figure 3-19 shows one example
of such classification using two GMM-s with 8 Gaussians each.
© A.G.Billard 2004 – Last Update March 2011