pdfcoffee

An introduction to AutoML
Feature construction has the goal of building new derived features, starting from
the basic ones. Frequently, this technique is used to allow better generalization and
to have a richer representation of the data.
Feature extraction aims at altering the original feature space by means of a
mapping function. This can be implemented in multiple ways; for instance, it can
use autoencoders (see Chapter 9, Autoencoders), PCA, or clustering (see Chapter 10,
Unsupervised Learning).
In short, feature engineering is an art based on intuition, trial and error, and a lot of
human experience. Modern AutoML engines aim to make the entire process more
automated, requiring less human intervention.
Automatic model generation
Model generation and hyperparameter tuning is the typical third macro-step of a
machine learning pipeline (see Figure 1).
Model generation consists of creating a suitable model for solving specific tasks.
For instance, you will probably use CNNs for visual recognition, and you will use
RNNs for either time series analysis or for sequences. Of course, many variants are
possible, each of which is manually crafted through a process of trial and error,
and works for very specific domains.
Hyperparameter tuning happens once the model is manually crafted. This process
is generally very computationally expensive and can significantly change the
quality of the results in a positive way. That's because tuning the hyperparameters
can help to optimize our model further.
Automatic model generation is the ultimate goal of any AutoML pipeline. How
can this be achieved? One approach consists in generating the model by combining
a set of primitive operations including convolution, pooling, concatenation, skip
connections, recurrent neural networks, autoencoders, and pretty much all the
deep learning models we have encountered throughout this book. These operations
constitute a (typically very large) search space to be explored, and the goal is to
make this exploration as efficient as possible. In AutoML jargon, the exploration is
called NAS, or Neural Architecture Search.
The seminal paper on AutoML [1] was produced in November 2016. The key idea
(see Figure 2) is to use reinforcement learning (RL, see Chapter 11, Reinforcement
Learning). An RNN acts as the controller and it generates the model descriptions of
candidate neural networks. RL is used to maximize the expected accuracy of the
generated architectures on a validation set.
[ 494 ]