MACHINE LEARNING TECHNIQUES - LASA

12
1.2.4 Exercise
For the following scenarios: 1) learning to drive a bicycle; 2) learning how to open a box with a
lever; 3) learning sign language (if lost on a Island with only deaf people), determine:
a) The variables at hand;
b) A good measure of performance,
c) A criteria of “good enough” optimality,
d) A threshold of sub-optimality (“too poor”)
e) The minimal time lag
1.3 Best Practices in ML
ML algorithms may be extremely sensitive to the particular choice of data you used for training
them. Ideally, you would like your training set to be sufficiently large to sample from the real
distribution of the data you try to estimate. In practice, this is not feasible. For instance, imagine
that you wish to train an algorithm to recognize human faces. When training the algorithm, you
may observe only a subset of all faces you may encounter in life, but you would still like your
algorithm to generalize a model of “human faces” from observing only 100 of them. If you provide
the algorithm with too many examples of the same subset of data of faces (e.g. by training the
algorithm only on faces of people with long hair), the algorithm may overfit, i.e. learn features
that are representatives of this part of the data but not of the global pattern you aimed at training
it on. In this case, the algorithm will end up recognizing only faces with long hair.
Each time an algorithm cannot detect correctly instances of a given class (e.g. human faces with
short hair) is called a false negative.
In addition to training the algorithm on a representative sample of the data to recognize, you
should also provide it with data that are counterexamples to this set to avoid what is called false
positives. For instance, in the above example, an algorithm that retained only the feature long
hair may incorrectly recognize a human face when presented with pictures of horses.
Finally, since ML algorithms are essentially looking at correlations across data, they may fit
spurious correlations, if provided with a poorly chosen set of training data. To ensure then that the
algorithm has generalized correctly over the subsets of training examples you used, a number of
good practices have been developed, which we review briefly below.
1.3.1 Training, validation and testing sets
Common practice to assess the validity of a Machine Learning algorithm is to measure its
performance against three data sets, the training, validation and testing sets. These three sets
are disjoint partitions of all the data at hand.
The training and validation sets are used for crossvalidation (see below) during the training
phase. In the above example when training an algorithm to recognize human faces, one would
typically choose a set of N different faces representative of all gender, ethnicities and other
fashionable additions (hair cuts, glasses, moustache, etc). One would then split this set into a
training set and a validation set, usually half/half or 1/3 rd – 2/3 rd ).
The testing set consists of a subset of the data which you would normally encounter once training
is completed. In the above example, this would consist of faces recorded by the camera of the
client to whom you will have sold the algorithm after training it in the laboratory.
© A.G.Billard 2004 – Last Update March 2011