advanced-algorithmic-trading

249
In quantitative finance applications it is often impossible to generate more data (in the case of
financial asset pricing series) as there is only one history to draw from.
The basic idea is to repeatedly sample, with replacement, data from the original training set
in order to produce multiple separate training sets. These are then used to allow meta-learner
methods to reduce the variance of their predictions, thus greatly improving their predictive
performance.
Two of the following ensemble techniques–bagging and Random Forests–make heavy use of
bootstrapping techniques, and they will now be discussed.
18.5.2 Bootstrap Aggregation
One of the main drawbacks of DTs is that they suffer from being high-variance estimators.
The addition of a small number of extra training points can dramatically alter the prediction
performance of a learned tree, despite the training data not changing to any great extent.
This is in contrast to a low-variance estimator, such as linear regression, which is not hugely
sensitive to the addition of extra points–at least those that are relatively close to the remaining
points.
One way to minimise this problem is to utilise a concept known as bootstrap aggregation
or bagging. The basic idea is to combine multiple leaners (such as DTs) all fitted on separate
bootstrapped samples and average their predictions in order to reduce the overall variance of
these predictions.
Why does this work? James et al (2013)[59] point out that if N independent and identically
distributed (iid) observations Z 1 , . . . , Z N are given, each with a variance of σ 2 then the variance
of the mean of the observations, ¯Z is σ 2 /N. That is, if the average of these observations is taken
the variance is reduced by a factor equal to the number of observations.
However in quantitative finance datasets it is often the case that there is only one set of
training data. This means it is difficult, if not impossible, to create multiple separate independent
training sets. This is where The Bootstrap comes in. It allows the generation of multiple training
sets that all use one larger set.
Using the notation from James et al (2013)[59] and the Random Forest article at Wikipedia[16],
if B separate bootstrapped samples of the training set are created, with separate model estimators
ˆf b (x), then averaging these leads to a low-variance estimator model, ˆf avg :
ˆf avg (x) = 1 B
B∑
ˆf b (x) (18.6)
b=1
This procedure is known as bagging[26]. It is highly applicable to DTs because they are
high-variance estimators. It provides one mechanism to reduce the variance substantially.
Carrying out bagging for DTs is straightforward. Hundreds or thousands of deeply-grown
(non-pruned) trees are created across B bootstrapped samples of the training data. They are
combined in the manner described above to significantly reduce the variance of the overall estimator.
One of the main benefits of bagging is that it is not possible to overfit the model solely by
increasing the number of bootstrap samples, B. This is also true for Random Forests but not
the method of Boosting.