www.allitebooks.com

Chapter 10
We only fit the X matrix once per value of n_clusters to (drastically) improve the
speed of this code:
inertia_scores = []
n_cluster_values = list(range(2, 20))
for n_clusters in n_cluster_values:
cur_inertia_scores = []
X = TfidfVectorizer(max_df=0.4).fit_transform(documents)
for i in range(10):
km = KMeans(n_clusters=n_clusters).fit(X)
cur_inertia_scores.append(km.inertia_)
inertia_scores.append(cur_inertia_scores)
The inertia_scores variable now contains a list of inertia scores for each
n_clusters value between 2 and 20. We can plot this to get a sense of how
this value interacts with n_clusters:
Overall, the value of the inertia should decrease with reducing improvement as
the number of clusters improves, which we can broadly see from these results.
The increase between values of 6 to 7 is due only to the randomness in selecting
the centroids, which directly affect how good the final results are. Despite this,
there is a general trend (in these results; your results may vary) that about 6
clusters is the last time a major improvement in the inertia occurred.
[ 227 ]