MACHINE LEARNING TECHNIQUES - LASA

116
5.10 Gaussian Process Classification
Generative versus Discriminative Approaches
To recall, there are two approaches to classification. One can either follow a so-called generative
p X,
Y that associates the set of
approach whereby one first learns the joint distribution ( )
datapoints X = { x 1 ,.... x
M
} to a set of labels { 1 ,..., M
Y y y }
= , where each class label denotes the
associated class numbering. Using the joint distribution, one can then compute the conditional
pY|
X to predict the class label for each datapoint. Alternatively, one can follow a
distribution ( )
discriminative approach in which one estimates directly the conditional distribution ( | )
pY X . In
the previous section, we showed how to perform regression using a Gaussian Process, it seems
hence intuitive to extend this to classification and to this end to take a discriminative approach.
Given that there are the generative and discriminative approaches, which one
should we prefer? This is perhaps the biggest question in classification, and we do
not believe that there is a right answer, as both ways of writing the joint p(y, x) are
correct. However, it is possible to identify some strengths and weaknesses of the
two approaches. The discriminative approach is appealing in that it is directly
modelling what we want, p(y|x). Also, density estimation for the class-conditional
distributions is a hard problem, particularly when x is high dimensional, so if we are
just interested in classification then the generative approach may mean that we are
trying to solve a harder problem than we need to. However, to deal with missing
input values, outliers and unlabelled data missing values points in a principled
fashion it is very helpful to have access to p(x), and this can be obtained from
p x = ∑ p y p x|
y in the
marginalizing out the class label y from the joint as ( ) ( ) ( )
generative approach. A further factor in the choice of a generative or discriminative
approach could also be which one is most conducive to the incorporation of any
prior information which is available. (Quote from C. E. Rasmussen & C. K. I.
Williams, Gaussian Processes for Machine Learning, the MIT Press, 2006.)
y
Linear Case
As for all other kernel methods send in this class, we will first start with the linear case and then
extend it to the non-linear case by exploiting the kernel trick.
One problem when performing multi-class classification is to compare the predictions of each
class. It would be easier if the output of the density could have a direct probabilistic interpretation.
In the simply binary classification problem where the label y takes either value +1 or value -1, i.e.
i i
∈− [ 1; + 1 ], = 1... , a probabilistic readout of the the conditional ( | )
i
y i M
easily by using, for instance, the logistic regressive model and compute:
p y x can be obtained
i
i
( 1| x )
p y
=+ =
1+
1
i
( x )
By extension, the probability of the label -1 is p( y i 1| x i ) 1 p( y i 1| x
i
)
e −
T
w
=− = − =+ .
(5.92)
© A.G.Billard 2004 – Last Update March 2011