Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

160 7. Principal Component Analysis and Factor AnalysisPCs are exact linear functions of x andhavetheformz = A ′ x.The factors, however, are not exact linear functions of x; instead x is definedas a linear function of f apart from an error term, and when the relationshipis reversed, it certainly does not lead to an exact relationship between fand x. Indeed, the fact that the expected value of x is a linear functionof f need not imply that the expected value of f is a linear function ofx (unless multivariate normal assumptions are made). Thus, the use ofPCs as initial factors may force the factors into an unnecessarily restrictivelinear framework. Because of the non-exactness of the relationship betweenf and x, the values of f, thefactor scores, mustbeestimated, and thereare several possible ways of doing this (see, for example, Bartholomew andKnott (1999, 3.23–3.25); Jackson (1991, Section 17.7); Lawley and Maxwell(1971, Chapter 8); Lewis-Beck (1994, Section II.6)).To summarize, there are many ways in which PCA and factor analysisdiffer from one another. Despite these differences, they both have the aim ofreducing the dimensionality of a vector of random variables. The use of PCsto find initial factor loadings, though having no firm justification in theory(except when Ψ = σ 2 I p as in Tipping and Bishop’s (1999a) model) willoften not be misleading in practice. In the special case where the elementsof Ψ are proportional to the diagonal elements of Σ, Gower (1966) showsthat the configuration of points produced by factor analysis will be similarto that found by PCA. In principal factor analysis, the results are equivalentto those of PCA if all (non-zero) elements of Ψ are identical (Rao, 1955).More generally, the coefficients found from PCA and the loadings foundfrom (orthogonal) factor analysis will often be very similar, although thiswill not hold unless all the elements of Ψ are of approximately the same size(Rao, 1955), which again relates to Tipping and Bishop’s (1999a) model.Schneeweiss and Mathes (1995) provide detailed theoretical comparisonsbetween factor analysis and PCA. Assuming the factor model (7.2.1), theycompare Λ with estimates of Λ obtained from PCA and from factor analysis.Comparisons are also made between f, the PC scores, and estimatesof f using factor analysis. General results are given, as well as comparisonsfor the special cases where m = 1 and where Σ = σ 2 I. The theorems, lemmasand corollaries given by Schneeweiss and Mathes provide conditionsunder which PCs and their loadings can be used as adequate surrogatesfor the common factors and their loadings. One simple set of conditionsis that p is large and that the elements of Ψ are small, although, unlikethe conventional factor model, Ψ need not be diagonal. Additionalconditions for closeness of factors and principal components are given bySchneeweiss (1997). Further, mainly theoretical, discussion of relationshipsbetween factor analysis and PCA appears in Ogasawara (2000).The results derived by Schneeweiss and Mathes (1995) and Schneeweiss(1997) are ‘population’ results, so that the ‘estimates’ referred to above