www.allitebooks.com

Working with Big Data
The final step is the reduce step, which takes a key value pair (the value in this case is
always a list) and produces a key value pair as a result. In our example, the key is the
word, the input list is the list of counts produced in the shuffle step, and the output
value is the sum of the counts:
def reduce_counts(word, list_of_counts):
return (word, sum(list_of_counts))
To see this in action, we can use the 20 newsgroups dataset, which is provided in
scikit-learn:
from sklearn.datasets import fetch_20newsgroups
dataset = fetch_20newsgroups(subset='train')
documents = dataset.data
We then apply our map step. We use enumerate here to automatically generate
document IDs for us. While they aren't important in this application, these keys
are important in other applications;
map_results = map(map_word_count, enumerate(documents))
The actual result here is just a generator, no actual counts have been produced.
That said, it is a generator that emits (word, count) pairs.
Next, we perform the shuffle step to sort these word counts:
shuffle_results = shuffle_words(map_results)
This, in essence is a MapReduce job; however, it is only running on a single thread,
meaning we aren't getting any benefit from the MapReduce data format. In the next
section, we will start using Hadoop, an open source provider of MapReduce, to start
to get the benefits from this type of paradigm.
Hadoop MapReduce
Hadoop is a set of open source tools from Apache that includes an implementation
of MapReduce. In many cases, it is the de facto implementation used by many.
The project is managed by the Apache group (who are responsible for the
famous web server).
[ 278 ]