www.allitebooks.com

Clustering News Articles
After this point, only slight improvements are made to the inertia, although it is
hard to be specific about vague criteria such as this. Looking for this type of pattern
is called the elbow rule, in that we are looking for an elbow-esque bend in the graph.
Some datasets have more pronounced elbows, but this feature isn't guaranteed to
even appear (some graphs may be smooth!).
Based on this analysis, we set n_clusters to be 6 and then rerun the algorithm:
n_clusters = 6
pipeline = Pipeline([('feature_extraction',
TfidfVectorizer(max_df=0.4)),
('clusterer', KMeans(n_clusters=n_clusters))
])
pipeline.fit(documents)
labels = pipeline.predict(documents)
Extracting topic information from clusters
Now we set our sights on the clusters in an attempt to discover the topics in each.
We first extract the term list from our feature extraction step:
terms = pipeline.named_steps['feature_extraction'].get_feature_names()
We also set up another counter for counting the size of each of our classes:
c = Counter(labels)
Iterating over each cluster, we print the size of the cluster as before. It is important
to keep in mind the sizes of the clusters when evaluating the results—some of the
clusters will only have one sample, and are therefore not indicative of a general
trend. The code is as follows:
for cluster_number in range(n_clusters):
print("Cluster {} contains {} samples".format(cluster_number,
c[cluster_number]))
Next (and still in the loop), we iterate over the most important terms for this cluster.
To do this, we take the five largest values from the centroid, which we get by finding
the features that have the highest values in the centroid itself. The code is as follows:
print(" Most important terms")
centroid = pipeline.named_steps['clusterer'].cluster_centers_
[cluster_number]
most_important = centroid.argsort()
[ 228 ]