Master Thesis - Department of Computer Science
Master Thesis - Department of Computer Science
Master Thesis - Department of Computer Science
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
4.3.2.1 Backward Selection<br />
The backward selection procedure starts from the full set <strong>of</strong> n features. Then, elim-<br />
inating one feature from n, all possible subsets <strong>of</strong> n-1 features are obtained and<br />
criterion values are evaluated for each <strong>of</strong> them. Then the subset corresponding to<br />
maximum value <strong>of</strong> the criterion is selected as the best subset containing n-1 features.<br />
This step repeats until the desired number <strong>of</strong> features are obtained. We evaluate<br />
all possible number <strong>of</strong> optimal features on a validation set and select one providing<br />
maximum separability. A toy example demonstrating backward selection with four<br />
features is shown in Figure 4.2(a).<br />
(1, 2, 3, 4)<br />
(2, 3, 4) (1, 3, 4) (1, 2, 4) (1, 2, 3)<br />
(3, 4) (1, 4) (1, 3)<br />
(a) Backward Selection<br />
4.3.2.2 Forward Selection<br />
1 2 3 4<br />
(1, 2) (2, 3)<br />
(1, 2, 3)<br />
(2, 3, 4)<br />
(2, 4)<br />
(b) Forward Selection<br />
Figure 4.2: Stepwise feature subset selection<br />
Forward selection on the other hand starts from the evaluation <strong>of</strong> individual features.<br />
For n number <strong>of</strong> features, forward selection evaluates the value <strong>of</strong> criterion on every<br />
individual feature and select one which provides maximum value. Now one more<br />
feature is added to this selected feature to form a subset <strong>of</strong> two. All possible such<br />
subsets are evaluated against the criterion. The subset providing the maximum value<br />
<strong>of</strong> the criterion is selected as the best subset <strong>of</strong> two features. This continues until<br />
the algorithm finds a subset <strong>of</strong> desired number <strong>of</strong> features. We compute all possible<br />
number <strong>of</strong> optimal features on a validation set and select the feature set providing<br />
maximum separability to evaluate on testing set. An example describing the operation<br />
<strong>of</strong> forward selection is shown in Figure 4.2(b).<br />
80