Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

10.3. Sensitivity and Stability 259tion for the second eigenvector and, in fact, has an influence nearly six timesas large as that of the second most influential observation. It is also themost influential on the first, third and fourth eigenvectors, showing that theperturbation caused by observation 16 to the second PC has a ‘knock-on’effect to other PCs in order to preserve orthogonality. Although observation16 is very influential on the eigenvectors, its effect is less marked on theeigenvalues. It has only the fifth highest influence on the second eigenvalue,though it is highest for the fourth eigenvalue, second highest for the first,and fourth highest for the third. It is clear that values of influence on eigenvaluesneed not mirror major changes in the structure of the eigenvectors,at least when dealing with correlation matrices.Having said that observation 16 is clearly the most influential, for eigenvectors,of the 28 observations in the data set, it should be noted that itsinfluence in absolute terms is not outstandingly large. In particular, the coefficientsrounded to one decimal place for the second PC when observation16 is omitted are0.2 0.1 − 0.4 0.8 − 0.1 − 0.3 − 0.2.The corresponding coefficients when all 28 observations are included are−0.0 − 0.2 − 0.2 0.9 − 0.1 − 0.0 − 0.0.Thus, when observation 16 is removed, the basic character of PC2 as mainlya measure of head size is unchanged, although the dominance of head sizein this component is reduced. The angle between the two vectors definingthe second PCs, with and without observation 16, is about 24 ◦ ,whichisperhaps larger than would be deduced from a quick inspection of the simplifiedcoefficients above. Pack et al. (1988) give a more thorough discussionof influence in the context of this data set, together with similar data setsmeasured on different groups of students (see also Brooks (1994)). A problemthat does not arise in the examples discussed here, but which does inPack et al.’s (1988) larger example, is the possibility that omission of anobservation causes switching or rotation of eigenvectors when consecutiveeigenvalues have similar magnitudes. What appear to be large changes ineigenvectors may be much smaller when a possible reordering of PCs inthe modified PCA is taken into account. Alternatively, a subspace of twoor more PCs may be virtually unchanged when an observation is deleted,but individual eigenvectors spanning that subspace can look quite different.Further discussion of these subtleties in the context of an example is givenby Pack et al. (1988).10.3 Sensitivity and StabilityRemoving a single observation and estimating its influence explores onetype of perturbation to a data set, but other perturbations are possible. In