Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

7.3. Comparisons Between Factor and Principal Component Analysis 159Σ by a small number of factors, whereas, conversely, PCA concentrates onthe diagonal elements of Σ.This leads to another difference between the two techniques concerningthe number of dimensions m which give an adequate representation of thep dimensional variable x. In PCA, if any individual variables are almostindependent of all other variables, then there will be a PC correspondingto each such variable, and that PC will be almost equivalent to the correspondingvariable. Such ‘single variable’ PCs are generally included if anadequate representation of x is required, as was discussed in Section 6.1.5.In contrast, a common factor in factor analysis must contribute to at leasttwo of the variables, so it is not possible to have a ‘single variable’ commonfactor. Instead, such factors appear as specific factors (error terms) and donot contribute to the dimensionality of the model. Thus, for a given setof data, the number of factors required for an adequate factor model is nolarger—and may be strictly smaller—than the number of PCs required toaccount for most of the variation in the data. If PCs are used as initialfactors, then the ideal choice of m is often less than that determined bythe rules of Section 6.1, which are designed for descriptive PCA. As notedseveral times in that Section, the different objectives underlying PCA andfactor analysis have led to confusing and inappropriate recommendationsin some studies with respect to the best choice of rules.The fact that a factor model concentrates on accounting for the offdiagonalelements, but not the diagonal elements, of Σ leads to variousmodifications of the idea of using the first m PCs to obtain initial estimatesof factor loadings. As the covariance matrix of the common factors’contribution to x is Σ − Ψ, it seems reasonable to use ‘PCs’ calculated forΣ − Ψ rather than Σ to construct initial estimates, leading to so-calledprincipal factor analysis. This will, of course, require estimates of Ψ, whichcan be found in various ways (see, for example, Rencher, 1998, Section 10.3;Rummel, 1970, Chapter 13), either once-and-for-all or iteratively, leadingto many different factor estimates. Many, though by no means all, of thedifferent varieties of factor analysis correspond to simply using differentestimates of Ψ in this type of ‘modified PC’ procedure. None of these estimateshas a much stronger claim to absolute validity than does the use ofthe PCs of Σ, although arguments have been put forward to justify variousdifferent estimates of Ψ.Another difference between PCA and (after rotation) factor analysis isthat changing m, the dimensionality of the model, can have much moredrastic effects on factor analysis than it does on PCA. In PCA, if m isincreased from m 1 to m 2 , then an additional (m 2 − m 1 ) PCs are included,but the original m 1 PCs are still present and unaffected. However, in factoranalysis an increase from m 1 to m 2 produces m 2 factors, none of which needbear any resemblance to the original m 1 factors.A final difference between PCs and common factors is that the formercan be calculated exactly from x, whereas the latter typically cannot. The