www.allitebooks.com

Chapter 12
The map function takes a key and value pair and returns a list of key+value pairs. The
keys for the input and output don't necessarily relate to each other. For example, for
a MapReduce program that performs a word count, the input key might be a sample
document's ID value, while the output key would be a given word. The input value
would be the text of the document and the output value would be the frequency of
each word:
from collections import defaultdict
def map_word_count(document_id, document):
We first count the frequency of each word. In this simplified example, we split the
document on whitespace to obtain the words, although there are better options:
counts = defaultdict(int)
for word in document.split():
counts[word] += 1
We then yield each of the word, count pairs. The word here is the key, with the count
being the value in MapReduce terms:
for word in counts:
yield (word, counts[word])
By using the word as the key, we can then perform a shuffle step, which groups all
of the values for each key:
def shuffle_words(results):
First, we aggregate the resulting counts for each word into a list of counts:
records = defaultdict(list)
We then iterate over all the results that were returned by the map function;
for results in results_generators:
for word, count in results:
records[word].append(count)
Next, we yield each of the words along with all the counts that were obtained
in our dataset:
for word in records:
yield (word, records[word])
[ 277 ]