MACHINE LEARNING TECHNIQUES - LASA

39
3 Clustering and Classification
Classification consists of categorizing data according to some criteria. This can be used as a
means to reduce the dimensionality of the dataset, prior to performing other form of data
processing on the constructed categories.
Classification is a key feature of most learning methods. In this chapter, we will first look at one
way to perform classification through data clustering. Then, we will look at a number of linear
classification methods that exploit eigenvalue decomposition such as those seen for PCA and
CCA in the previous chapter. All of these methods are in effect supervised as they rely on an
apriori estimate of the number of classes or clusters. Note that we will see other methods that can
be used to perform classification in other chapters of these lecture notes. These are for instance
feed-forward neural networks, or Hidden Markov Models. As it is often the case, machine learning
methods may be used in various ways and there is thus not a single way to classify these!
3.1 Clustering Techniques
Clustering is a type of multivariate statistical analysis also known as cluster analysis,
unsupervised classification analysis, or numerical taxonomy.
Looking at Figure 3-1, one can easily see that the points group naturally into 3 separate clusters,
albeit of different shape and size. While it is obvious to see this when looking at the graph, it
would become less so obvious if you were to look at the coordinates of these points.
Figure 3-1: Example of a dataset that form 3 clusters
There are many ways to cluster data depending on the criteria required for the task. One can, for
instance, group the data or segment the data into clusters. Clusters can be either globular or
convex, such that any line you can draw between two cluster members stays inside the
boundaries of the cluster, or non-globular or concave, i.e. taking any shape. Clusters can contain
an equal number of data, or be of equal size (distance between the data). Central to all of the
goals of cluster analysis is the notion of degree of similarity (or dissimilarity) between the
individual objects being clustered. This notion determines the type of clusters that will be formed.
In this chapter, we will see different methods of clustering that belong to two major classes of
clustering algorithms, namely hierarchical clustering and partitioning algorithms. We will
discuss the limitation of such methods. In particular, we will look at a generalization of K-means
clustering, namely soft k-means and mixture of Gaussians that enable to overcome some but
no all limitations of simple K-means.
© A.G.Billard 2004 – Last Update March 2011