www.allitebooks.com

Working with Big Data
We start by creating a function that takes two numbers and adds them together.
def add(a, b):
return a + b
We then perform the reduce. The signature of reduce is reduce(function, sequence,
and initial), where the function is applied at each step to the sequence. In the first
step, the initial value is used as the first value, rather than the first element of the list:
from functools import reduce
print(reduce(add, sums, 0))
The result, 25, is the sum of each of the values in the sums list and is consequently
the sum of each of the elements in the original array.
The preceding code is equal to the following:
initial = 0
current_result = initial
for element in sums:
current_result = add(current_result, element)
In this trivial example, our code can be greatly simplified, but the real gains come
from distributing the computation. For instance, if we have a million sublists
and each of those sublists contained a million elements, we can distribute this
computation over many computers.
In order to do this, we distribute the map step. For each of the elements in our list,
we send it, along with a description of our function, to a computer. This computer
then returns the result to our main computer (the master).
The master then sends the result to a computer for the reduce step. In our example
of a million sublists, we would send a million jobs to different computers (the same
computer may be reused after it completes our first job). The returned result would
be just a single list of a million numbers, which we then compute the sum of.
The result is that no computer ever needed to store more than a million numbers,
despite our original data having a trillion numbers in it.
A word count example
The implementation of MapReduce is a little more complex than just using a map
and reduce step. Both steps are invoked using keys, which allows for the separation
of data and tracking of values.
[ 276 ]