www.allitebooks.com

Classifying with scikit-learn Estimators
Running the algorithm
In our earlier experiments, we set aside a portion of the dataset as a testing set,
with the rest being the training set. We train our algorithm on the training set and
evaluate how effective it will be based on the testing set. However, what happens if
we get lucky and choose an easy testing set? Alternatively, what if it was particularly
troublesome? We can discard a good model due to poor results resulting from such
an "unlucky" split of our data.
The cross-fold validation framework is a way to address the problem of choosing a
testing set and a standard methodology in data mining. The process works by doing
a number of experiments with different training and testing splits, but using each
sample in a testing set only once. The procedure is as follows:
1. Split the entire dataset into a number of sections called folds.
2. For each fold in the dataset, execute the following steps:
° ° Set that fold aside as the current testing set
° ° Train the algorithm on the remaining folds
° ° Evaluate on the current testing set
3. Report on all the evaluation scores, including the average score.
4. In this process, each sample is used in the testing set only once.
This reduces (but doesn't completely eliminate) the likelihood of
choosing lucky testing sets.
Throughout this book, the code examples build upon each
other within a chapter. Each chapter's code should be
entered into the same IPython Notebook, unless
otherwise specified.
The scikit-learn library contains a number of cross fold validation methods.
A helper function is given that performs the preceding procedure. We can import
it now in our IPython Notebook:
from sklearn.cross_validation import cross_val_score
By default, cross_val_score uses a specific methodology
called Stratified K Fold to split the dataset into folds. This
creates folds that have approximately the same proportion
of classes in each fold, again reducing the likelihood of
choosing poor folds. This is a great default, so we won't
mess with it right now.
[ 32 ]