pdfcoffee

Chapter 13
Traditional machine learning requires you to have a centralized repository for
training data either on your desktop, or in your datacenter, or in the cloud. Federated
learning pushes the training phase at the edge by distributing the computation
among millions of mobile devices. These devices are ephemeral in that they are
not always available for the learning process and they can disappear silently (for
instance, a mobile phone can be switched off all of a sudden). The key idea is
to leverage the CPUs and the GPU of each mobile phone that is made available
for an FL computation. Each mobile device forming a part of a distributed FL
training downloads a (pretrained) model from a central server and it performs
local optimization based on the local training data collected on each specific
mobile device. This process is similar to the transfer learning process (see Chapter
5, Advanced Convolutional Neural Networks), but it is distributed at the edge. Each
locally updated model is then sent back by millions of edge devices to a central
server to build an averaged shared model.
Of course, there are many issues to be considered. Let's review them:
1. Battery usage: Each mobile device that is part of an FL computation
should save as much as possible on local battery usage.
2. Encrypted communication: Each mobile device belonging to an FL
computation has to use encrypted communication with the central server to
update the locally built model.
3. Efficient communication: Typically, deep learning models are optimized
with optimization algorithms such as SGD (see Chapter 1, Neural Network
Foundations with TensorFlow 2.0, and Chapter 15, The Math Behind Deep
Learning). However, FL works with millions of devices and there is
therefore a strong need to minimize the communication patterns. Google
introduced a Federated Averaging algorithm [8], which is reported to reduce
the amount of communication 10x-100x when compared with vanilla SGD.
Plus, compression techniques [9] reduce the communication costs by an
additional 100x with random rotations and quantization.
4. Ensure user privacy: This is probably the most important point. All local
training data acquired at the edge must stay at the edge. This means that the
training data acquired on a mobile device cannot be sent to a central server.
Equally important, any user behavior learned in locally trained models must
be anonymized so that it is not possible to understand any specific action
performed by specific individuals.
Figure 11 shows a typical FL architecture (source [10]). An FL Server sends a model
and a training plan to millions of devices. The training plan includes information on
how frequently updates are expected and other metadata.
[ 475 ]