Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

8.1. Principal Component Regression 171This expression gives insight into how multicollinearities produce largevariances for the elements of ˆβ. If a multicollinearity exists, then it appearsas a PC with very small variance (see also Sections 3.4 and 10.1); in otherwords, the later PCs have very small values of l k (the variance of the kthPC is l k /(n−1) in the present notation), and hence very large values of l −1k .Thus (8.1.9) shows that any predictor variable having moderate or largecoefficients in any of the PCs associated with very small eigenvalues willhave a very large variance.One way of reducing this effect is to delete the terms from (8.1.8) thatcorrespond to very small l k , leading to an estimator˜β =m∑k=1l −1k a ka ′ kX ′ y, (8.1.10)where l m+1 ,l m+2 ,...,l p are the very small eigenvalues. This is equivalentto setting the last (p − m) elements of γ equal to zero.Then the variance-covariance matrix V (˜β) for˜β isSubstitutingσ 2m∑j=1l −1ja j a ′ jX ′ XX ′ X =m∑k=1p∑l h a h a ′ hh=1l −1k a ka ′ k.from the spectral decomposition of X ′ X,wehaveV (˜β) =σ 2p∑m∑m∑h=1 j=1 k=1l h l −1j l −1k a ja ′ ja h a ′ ha k a ′ k.Because the vectors a h ,h=1, 2,...,p are orthonormal, the only non-zeroterms in the triple summation occur when h = j = k, sothatV (˜β) =σ 2m∑k=1l −1k a ka ′ k (8.1.11)If none of the first m eigenvalues l k is very small, then none of the variancesgiven by the diagonal elements of (8.1.11) will be large.The decrease in variance for the estimator ˜β given by (8.1.10), comparedwith the variance of ˆβ, is achieved at the expense of introducing bias intothe estimator ˜β. This follows because˜β = ˆβ −p∑k=m+1l −1k a ka ′ kX ′ y, E(ˆβ) =β,