www.allitebooks.com

Social Media Insight Using Naive Bayes
Then for each tweet in the search results, we save it to our file in the same way we
did when we were collecting the dataset originally:
for tweet in search_results:
if 'text' in tweet:
output_file.write(json.dumps(tweet))
output_file.write("\n\n")
As a final step here (and still under the preceding if block), we want to store the
labeling of this tweet. We can do this using the label_mapping dictionary we
created before, looking up the tweet ID. The code is as follows:
actual_labels.append(label_mapping[tweet['id']])
Run the previous cell and the code will collect all of the tweets for you. If you created
a really big dataset, this may take a while—Twitter does rate-limit requests. As a
final step here, save the actual_labels to our classes file:
with open(labels_filename, 'w') as outf:
json.dump(actual_labels, outf)
Text transformers
Now that we have our dataset, how are we going to perform data mining on it?
Text-based datasets include books, essays, websites, manuscripts, programming
code, and other forms of written expression. All of the algorithms we have seen so
far deal with numerical or categorical features, so how do we convert our text into a
format that the algorithm can deal with?
There are a number of measurements that could be taken. For instance, average
word and average sentence length are used to predict the readability of a document.
However, there are lots of feature types such as word occurrence which we will
now investigate.
Bag-of-words
One of the simplest but highly effective models is to simply count each word in the
dataset. We create a matrix, where each row represents a document in our dataset
and each column represents a word. The value of the cell is the frequency of that
word in the document.
[ 118 ]