MACHINE LEARNING TECHNIQUES - LASA

45
Disadvantages of hierarchical clustering are related to:
• Vagueness of termination criteria
• The fact that most hierarchical algorithms do not revisit once constructed (intermediate)
clusters with the purpose of their improvement
Exercise:
Determine the optimal distance metric for two examples, in which the data are either not linearly
separable or linearly separable.
3.1.2 K-means clustering
K-Means clustering generates a number K of disjoint, flat (non-hierarchical)
k
clusters k = K so as to minimize the sum-of-squares criterion
C , 1,... ,
J
( ,... )
µ µ = x −µ
1
K
K i k
k = 1
k
i∈C
2
∑∑ (3.2)
where x i
is a vector representing the i th data point and µ is the geometric centroid of the data
k
points associated to cluster C . K-means is well suited to generating globular clusters. The K-
Means method is numerical, unsupervised, non-deterministic and iterative. In general, the
algorithm is not guaranteed to converge to a global minimum of J.
k
K-Means Algorithm:
1 Initialization: Pick K arbitrary centroids and set their geometric means µ ,..., 1
µ to
K
random values.
2 Calculate the distance dx ( , µ ) from each data point i to each centroid k .
3 Assignment Step: Assign the responsibility
i
k
i
r
k
of each data point i to its “closest”
centroid k
i
(E-step). If a tie happens (i.e. two centroids are equidistant to a data
point, one assigns the data point to the smallest winning centroid).
{ ( )}
k arg min d x , µ
= (3.3)
i i k
k
i
k
⎧⎧ 1 if k = k
ri
= ⎨⎨
⎩⎩ 0 otherwise
4 Update Step: Adjust the centroids to be the means of all data points assigned to
them (M-step)
µ =
k
∑
i
∑
i
rx
k
i
r
k
i
i
(3.4)
© A.G.Billard 2004 – Last Update March 2011