MACHINE LEARNING TECHNIQUES - LASA

13
1.3.2 Crossvalidation
To ensure that the splitting of data into training and validation sets was not a lucky split leading to
optimal but non representative performance, one must apply crossvalidation. “Crossvalidation
refers to the practice of confirming an experimental finding by repeating the experiment using an
independent assay technique” (Wikipedia). In ML, this consists of splitting the dataset several
times at random into training and validation sets. The performance of the algorithms is then
measured on the whole set of experiments, e.g. by taking the mean and standard deviation of the
error when applying the algorithm on each set. The standard deviation is important as it gives a
measure of potential discrepancies in the structure of your data set. The crossvalidation phase is
used to fine tune parameters of the algorithm (e.g. thresholds, priors, learning rates, etc) to
ensure optimal performance.
1.3.3 Performance Measures in ML
1.3.3.1 Ground Truth
Comparing the performance of a novel method to existing ones or trivial baselines is crucial.
Whenever possible, one will use the ground truth as a means to compare performance of the
system during training.
In classification, the ground truth can be obtained for instance from manually labeling each data
with the appropriate class label. If the choice of class label may be subjective (e.g. labeling facial
expressions of happiness, sadness, etc), one may ask several raters to label the picture and one
may take the average score across raters for the “true” label.
1.3.3.2 ROC
Graphical representations such as the Receiver Operating Characteristic, so called ROC curve
provide an interesting way to quickly compare algorithms independently of the choice of
parameters.
The ROC curve is used in binary classification algorithms. It balances the so-called true positives
(TP), i.e. instances when the algorithm classified correctly a datapoint to (y=+1) and the false
positives (FP), i.e. instances when the algorithm classified incorrectly datapoints to (y=+1).
Conversely, false positives (FP) and false negatives (FN) correspond to the above two cases with
classification (y=-1).
The ROC curve plots the fraction of true positives and false positives over the total number of
samples of class y=+1 in the dataset. Each point on the curve corresponds to a different value of
the classifier’s parameter (usually a threshold; e.g. a threshold on Bayes’ classification). Typical
ROC curve is drawn below.
© A.G.Billard 2004 – Last Update March 2011