www.allitebooks.com

Chapter 12
The first function is the mapper function for the first step. The goal of this function
is to take each blog post, get all the words in that post, and then note the occurrence.
We want the frequencies of the words, so we will return 1 / len(all_words),
which allows us to later sum the values for frequencies. The computation here isn't
exactly correct—we need to also normalize for the number of documents. In this
dataset, however, the class sizes are the same, so we can conveniently ignore this
with little impact on our final version.
We also output the gender of the post's author, as we will need that later:
def extract_words_mapping(self, key, value):
tokens = value.split()
gender = eval(tokens[0])
blog_post = eval(" ".join(tokens[1:]))
all_words = word_search_re.findall(blog_post)
all_words = [word.lower() for word in all_words]
all_words = word_search_re.findall(blog_post)
all_words = [word.lower() for word in all_words]
for word in all_words:
yield (gender, word), 1. / len(all_words)
We used eval in the preceding code to simplify the parsing of the blog
posts from the file, for this example. This is not recommended. Instead,
use a format such as JSON to properly store and parse the data from
the files. A malicious use with access to the dataset can insert code into
these tokens and have that code run on your server.
In the reducer for the first step, we sum the frequencies for each gender and word
pair. We also change the key to be the word, rather than the combination, as this
allows us to search by word when we use the final trained model (although, we still
need to output the gender for later use);
def reducer_count_words(self, key, frequencies):
s = sum(frequencies)
gender, word = key
yield word, (gender, s)
The final step doesn't need a mapper function, so we don't add one. The data will
pass straight through as a type of identity mapper. The reducer, however, will
combine frequencies for each gender under the given word and then output the
word and frequency dictionary.
[ 287 ]