pdfcoffee

An introduction to AutoML
This Efficient Neural Architecture Search (ENAS) delivers strong empirical
performance using significantly fewer GPU-hours compared with all existing
automatic model design approaches, and notably, is 1000x less computationally
expensive than standard Neural Architecture Search (in 2018). Here, the primary
ENAS goal is to reduce the search space via hierarchical composition. Variants of
the cell-based approach have been proposed, including pure hierarchical methods
where higher-level cells are generated by incorporating lower-level cells iteratively.
Still considering NAS, a completely different idea is to use transfer learning
(see Chapter 5, Advanced Convolutional Neural Networks) to transfer the learning of an
existing neural network into a new neural network in order to speed up the design
[3]. In other words, we want to use transfer learning in AutoML.
Another approach is based on Genetic Programming (GP) and Evolutionary
algorithms (EAs) where the basic operations constituting the model search space are
encoded into a suitable representation and then this encoding is gradually mutated
to progressively better models in a way that resembles the genetic evolution of
living beings [4].
Hyperparameter tuning consists of finding the optimal combination of
hyperparameters both related to learning optimization (batch size, learning rate,
and so on) and model-specific ones (kernel size; number of feature maps and so on
for CNNs; number of neurons for dense or autoencoder networks, and so on). Again,
the search space can be extremely large. There are two approaches generally used:
grid search and random search.
Grid search divides the search space into a discrete grid of values and tests all the
possible combinations in the grid. For instance, if there are three hyperparameters
and a grid with only two candidate values for each of them, then a total of 2 × 3 = 6
combinations must be checked. There are also hierarchical variants of grid search,
which progressively refine the grid for regions of the search space and provide
better results. The key idea is to use a coarse grid first, and after finding a better grid
region, implement a finer grid search on that region.
Random search performs a random sampling of the parameter search space, and this
simple approach has been proven to work very well in many situations [5].
Now that we have briefly discussed the fundamentals we will do quite a bit of
hands-on work on Google Cloud. Let's start.
[ 496 ]