Master Thesis - Department of Computer Science

4.3.2.1 Backward Selection
The backward selection procedure starts from the full set of n features. Then, elim-
inating one feature from n, all possible subsets of n-1 features are obtained and
criterion values are evaluated for each of them. Then the subset corresponding to
maximum value of the criterion is selected as the best subset containing n-1 features.
This step repeats until the desired number of features are obtained. We evaluate
all possible number of optimal features on a validation set and select one providing
maximum separability. A toy example demonstrating backward selection with four
features is shown in Figure 4.2(a).
(1, 2, 3, 4)
(2, 3, 4) (1, 3, 4) (1, 2, 4) (1, 2, 3)
(3, 4) (1, 4) (1, 3)
(a) Backward Selection
4.3.2.2 Forward Selection
1 2 3 4
(1, 2) (2, 3)
(1, 2, 3)
(2, 3, 4)
(2, 4)
(b) Forward Selection
Figure 4.2: Stepwise feature subset selection
Forward selection on the other hand starts from the evaluation of individual features.
For n number of features, forward selection evaluates the value of criterion on every
individual feature and select one which provides maximum value. Now one more
feature is added to this selected feature to form a subset of two. All possible such
subsets are evaluated against the criterion. The subset providing the maximum value
of the criterion is selected as the best subset of two features. This continues until
the algorithm finds a subset of desired number of features. We compute all possible
number of optimal features on a validation set and select the feature set providing
maximum separability to evaluate on testing set. An example describing the operation
of forward selection is shown in Figure 4.2(b).
80