www.allitebooks.com

Social Media Insight Using Naive Bayes
Now we create a dataset by looping over both the tweets and labels at the same
time and saving those in a list:
dataset = [(tweet['id'], label) for tweet, label in zip(tweets,
labels)]
Finally, we save the results in our file:
with open(replicable_dataset, 'w') as outf:
json.dump(dataset, outf)
Now that we have the tweet IDs and labels saved, we can recreate the original
dataset. If you are looking to recreate the dataset I used for this chapter, it can be
found in the code bundle that comes with this book.
Loading the preceding dataset is not difficult but it can take some time. Start a
new IPython Notebook and set the dataset, label, and tweet ID filenames as before.
I've adjusted the filenames here to ensure that you don't overwrite your previously
collected dataset, but feel free to change these if you want. The code is as follows:
import os
tweet_filename = os.path.join(os.path.expanduser("~"), "Data",
"twitter", "replicable_python_tweets.json")
labels_filename = os.path.join(os.path.expanduser("~"), "Data",
"twitter", "replicable_python_classes.json")
replicable_dataset = os.path.join(os.path.expanduser("~"),
"Data", "twitter", "replicable_dataset.json")
Then load the tweet IDs from the file using JSON:
import json
with open(replicable_dataset) as inf:
tweet_ids = json.load(inf)
Saving the labels is very easy. We just iterate through this dataset and extract
the IDs. We could do this quite easily with just two lines of code (open file and save
tweets). However, we can't guarantee that we will get all the tweets we are after
(for example, some may have been changed to private since collecting the dataset)
and therefore the labels will be incorrectly indexed against the data.
As an example, I tried to recreate the dataset just one day after collecting them and
already two of the tweets were missing (they might be deleted or made private by
the user). For this reason, it is important to only print out the labels that we need.
To do this, we first create an empty actual labels list to store the labels for tweets
that we actually recover from twitter, and then create a dictionary mapping the
tweet IDs to the labels.
[ 116 ]