Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

30 3. Properties of Sample Principal ComponentsThe seventh section then goes on to show how these distributions may beused to make statistical inferences about the population PCs, based onsample PCs.Section 3.8 demonstrates how the approximate structure and variancesof PCs can sometimes be deduced from patterns in the covariance or correlationmatrix. Finally, in Section 3.9 we discuss models that have beenproposed for PCA. The material could equally well have been included inChapter 2, but because the idea of maximum likelihood estimation arisesin some of the models we include it in the present chapter.3.1 Optimal Algebraic Properties of SamplePrincipal ComponentsBefore looking at the properties themselves, we need to establish somenotation. Suppose that we have n independent observations on the p-element random vector x; denote these n observations by x 1 , x 2 ,...,x n .Let ˜z i1 = a ′ 1x i ,i=1, 2,...,n, and choose the vector of coefficients a ′ 1 tomaximize the sample variance1n − 1n∑(˜z i1 − ¯z 1 ) 2i=1subject to the normalization constraint a ′ 1a 1 = 1. Next let ˜z i2 = a ′ 2x i ,i=1, 2,...,n,andchoosea ′ 2 to maximize the sample variance of the ˜z i2 subjectto the normalization constraint a ′ 2a 2 = 1, and subject also to the ˜z i2 beinguncorrelated with the ˜z i1 in the sample. Continuing this process in anobvious manner, we have a sample version of the definition of PCs given inSection 1.1. Thus a ′ kx is defined as the kth sample PC, k =1, 2,...,p,and˜z ik is the score for the ith observation on the kth PC. If the derivation inSection 1.1 is followed through, but with sample variances and covariancesreplacing population quantities, then it turns out that the sample varianceof the PC scores for the kth sample PC is l k ,thekth largest eigenvalue of thesample covariance matrix S for x 1 , x 2 ,...,x n ,anda k is the correspondingeigenvector for k =1, 2,...,p.Define the (n × p) matrices ˜X and ˜Z to have (i, k)th elements equal tothe value of the kth element ˜x ik of x i , and to ˜z ik , respectively. Then ˜Z and˜X are related by ˜Z = ˜XA, where A is the (p × p) orthogonal matrix whosekth column is a k .If the mean of each element of x is known to be zero, then S = 1 ˜X ′ n˜X.It is far more usual for the mean of x to be unknown, and in this case the(j, k)th element of S is1n − 1n∑(˜x ij − ¯x j )(˜x ik − ¯x k ),i=1