Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

158 7. Principal Component Analysis and Factor Analysis7.3 Comparisons and Contrasts Between FactorAnalysis and Principal Component AnalysisAs mentioned in Section 7.1 a major distinction between factor analysisand PCA is that there is a definite model underlying factor analysis, butfor most purposes no model is assumed in PCA. Section 7.2 concluded bydescribing the most common way in which PCs are used in factor analysis.Further connections and contrasts between the two techniques are discussedin the present section, but first we revisit the ‘models’ that have beenproposed for PCA. Recall from Section 3.9 that Tipping and Bishop (1999a)describe a model in which x has covariance matrix BB ′ + σ 2 I p , where B isa(p × q) matrix. Identifying B with Λ, andq with m, it is clear that thismodel is equivalent to a special case of equation (7.2.1) in which Ψ = σ 2 I p ,so that all p specific variances are equal.De Leeuw (1986) refers to a generalization of Tipping and Bishop’s(1999a) model, in which σ 2 I p is replaced by a general covariance matrix forthe error terms in the model, as the (random factor score) factor analysismodel. This model is also discussed by Roweis (1997). A related model,in which the factors are assumed to be fixed rather than random, correspondsto Caussinus’s (1986) fixed effects model, which he also calls the‘fixed factor scores model.’ In such models, variability amongst individualsis mainly due to different means rather than to individuals’ covariancestructure, so they are distinctly different from the usual factor analysisframework.Both factor analysis and PCA can be thought of as trying to representsome aspect of the covariance matrix Σ (or correlation matrix) as wellas possible, but PCA concentrates on the diagonal elements, whereas infactor analysis the interest is in the off-diagonal elements. To justify thisstatement, consider first PCA. The objective is to maximize ∑ mk=1 var(z k)or, as ∑ pk=1 var(z k)= ∑ pj=1 var(x j), to account for as much as possibleof the sum of diagonal elements of Σ. As discussed after Property A3 inSection 2.1, the first m PCs will in addition often do a good job of explainingthe off-diagonal elements of Σ, which means that PCs can frequentlyprovide an adequate initial solution in a factor analysis. However, this isnot the stated purpose of PCA and will not hold universally. Turning nowto factor analysis, consider the factor model (7.1.2) and the correspondingequation (7.2.1) for Σ. It is seen that, as Ψ is diagonal, the common factorterm Λf in (7.1.2) accounts completely for the off -diagonal elementsof Σ in the perfect factor model, but there is no compulsion for the diagonalelements to be well explained by the common factors. The elements,ψ j ,j=1, 2,...,p,ofΨ will all be low if all of the variables have considerablecommon variation, but if a variable x j is almost independent of allother variables, then ψ j =var(e j ) will be almost as large as var(x j ). Thus,factor analysis concentrates on explaining only the off-diagonal elements of