Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

6.4. Examples Illustrating Variable Selection 149as they account for 87.8%, 93.3%, respectively, of the total variation. Thethird and fourth eigenvalues are 0.96, 0.68 so that a cut-off of l ∗ =0.70gives m = 3, but l 4 is so close to 0.70 that caution suggests m = 4. Suchconservatism is particularly appropriate for small sample sizes, where samplingvariation may be substantial. As in the previous example, Jolliffe(1973) found that the inclusion of a fourth variable produced a markedimprovement in reproducing some of the results given by all 18 variables.McCabe (1982) also indicated that m = 3 or 4 is appropriate.The subsets chosen in Table 6.5 overlap less than in the previous example,and McCabe’s subsets change noticeably in going from m =3tom =4.However, there is still substantial agreement; for example, variable 1 isa member of all but one of the selected subsets and variable 13 is alsoselected by all four methods, whereas variables {2, 6, 9, 11, 12, 15, 18} arenot selected at all.Of the variables that are chosen by all four methods, variable 1 is ‘homicide,’which dominates the third PC and is the only crime whose occurrenceshows no evidence of serial correlation during the period 1950–63. Becauseits behaviour is different from that of all the other variables, it is importantthat it should be retained in any subset that seeks to account for mostof the variation in x. Variable 13 (assault) is also atypical of the generalupward trend—it actually decreased between 1950 and 1963.The values of the criteria (6.3.4) and (6.3.5) for Jolliffe’s and McCabe’ssubsets are closer to optimality and less erratic than in the earlier example.No chosen subset does worse with respect to (6.3.5) than 0.925 for 3variables and 0.964 for 4 variables, compared to optimal values of 0.942,0.970 respectively. The behaviour with respect to (6.3.4) is less good, butfar less erratic than in the previous example.In addition to the examples given here, Al-Kandari (1998), Cadima andJolliffe (2001), Gonzalez et al. (1990), Jolliffe (1973), King and Jackson(1999) and McCabe (1982, 1984) all give further illustrations of variableselection based on PCs. Krzanowski (1987b) looks at variable selection forthe alate adelges data set of Section 6.4.1, but in the context of preservinggroup structure. We discuss this further in Chapter 9.