www.allitebooks.com

Working with Big Data
Then, make a folder for our test set:
mkdir blogs_test
Move any file starting with a 6 or 7 into the test set, from the train set:
cp blogs/6* blogs_test/
cp blogs/7* blogs_test/
We will rerun the blog extraction on all files in the training set. However, this is
a large computation that is better suited to cloud infrastructure than our system.
For this reason, we will now move the parsing job to Amazon's infrastructure.
Run the following on the command line, as you did before. The only difference is
that we train on a different folder of input files. Before you run the following code,
delete all files in the blog posts and models folders:
python extract_posts.py ~/Data/blogs_train --output-dir=/home/bob/
Data/blogposts –no-output
python nb_train.py ~/Data/blogposts/ --output-dir=/home/bob/models/
--no-output
The code here will take quite a bit longer to run.
We will test on any blog file in our test set. To get the files, we need to extract
them. We will use the extract_posts.py MapReduce job, but store the files
in a separate folder:
python extract_posts.py ~/Data/blogs_test --output-dir=/home/bob/Data/
blogposts_testing –no-output
Back in the IPython Notebook, we list all the outputted testing files:
testing_folder = os.path.join(os.path.expanduser("~"), "Data",
"blogposts_testing")
testing_filenames = []
for filename in os.listdir(testing_folder):
testing_filenames.append(os.path.join(testing_folder, filename))
For each of these files, we extract the gender and document and then call the predict
function. We do this in a generator, as there are a lot of documents, and we don't
want to use too much memory. The generator yields the actual gender and the
predicted gender:
def nb_predict_many(model, input_filename):
with open(input_filename) as inf:
# remove leading and trailing whitespace
[ 292 ]