MACHINE LEARNING TECHNIQUES - LASA

56
3.2 Linear Classifiers
In this section, we will consider solely ways to provide a linear classification of data. Non-linear
methods for classification such as ANN with backpropagation and Support Vector Machine will be
covered later on, in these lecture notes.
Linear Discriminant Analysis and the related Fisher's linear discriminant are methods to find the
linear combination of features (projections of the data) that best separate two or more classes of
objects. The resulting combination may be used as a linear classifier. We describe these next.
3.2.1 Linear Discriminant Analysis
Linear Discriminant Analysis (LDA) combines concepts of PCA and clustering to determine
projections along which a dataset can be best separated into two distinct classes.
M×
N
Consider a data matrix composed of M N-dimensional data points, i.e. X ∈° . LDA aims to
M×
P
i
find a linear transformation A∈° that maps each column x of X, for i= 1,... M (these are N-
dimensional vectors) into a corresponding q-dimensional vector y
i
. That is,
( )
i i i
: N
q
A x ∈ → y = Ax ∈ q≤N
° ° (3.22)
Let us further assume that the data in X is partitioned into K classes{ } , where each class
k = 1
k
C contains
K
k
n data points and
∑
k = 1
n
k
= M
. LDA aims to find the optimal transformation A such
that the class structure of the original high-dimensional space is preserved in the low-dimensional
space.
In general, if each class is tightly grouped, but well separated from the other classes, the quality
of the cluster is considered to be high. In discriminant analysis, two scatter matrices, called
within-class ( S ) and between-class ( S
w
b
) matrices, are defined to quantify the quality of the
clusters as follows:
where
k 1 1
µ = , µ =
n
N
w
K
i k i k
∑∑( µ )( µ )
S = x − x −
S
∑ ∑∑
k = 1 x ∈C
k
k
x∈C x∈C
k = 1
i
k
( µ µ )( µ µ )
K
k
b
= ∑ n
k
−
k
−
k = 1
K
k
T
T
C
k
K
(3.23)
are, respectively, the mean of the k-th class and the
global mean. An implicit assumption of LDA is that all classes have equal class covariance
(otherwise, the elements of the within-class matrix should be normalized by the covariance on the
set of data points of that class).
© A.G.Billard 2004 – Last Update March 2011