Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

178 8. Principal Components in Regression AnalysisPCs directly, but there are various relationships between PC regression andother biased regression methods which will be briefly discussed.Consider first ridge regression, which was described by Hoerl and Kennard(1970a,b) and which has since been the subject of much debate inthe statistical literature. The estimator of β using the technique can bewritten, among other ways, asp∑ˆβ R = (l k + κ) −1 a k a ′ kX ′ y,k=1where κ is some fixed positive constant and the other terms in the expressionhave the same meaning as in (8.1.8). The variance-covariance matrixof ˆβ R , is equal top∑l k (l k + κ) −2 a k a ′ k.σ 2k=1Thus, ridge regression estimators have rather similar expressions to thosefor least squares and PC estimators, but variance reduction is achievednot by deleting components, but by reducing the weight given to the latercomponents. A generalization of ridge regression has p constants κ k , k =1, 2,...,p that must be chosen, rather than a single constant κ.A modification of PC regression, due to Marquardt (1970) uses a similar,but more restricted, idea. Here a PC regression estimator of the form(8.1.10) is adapted so that M includes the first m integers, excludes theintegers m +2,m+3,...,p, but includes the term corresponding to integer(m + 1) with a weighting less than unity. Detailed discussion of suchestimators is given by Marquardt (1970).Ridge regression estimators ‘shrink’ the least squares estimators towardsthe origin, and so are similar in effect to the shrinkage estimators proposedby Stein (1960) and Sclove (1968). These latter estimators start withthe idea of shrinking some or all of the elements of ˆγ (or ˆβ) using argumentsbased on loss functions, admissibility and prior information; choiceof shrinkage constants is based on optimization of MSE criteria. Partialleast squares regression is sometimes viewed as another class of shrinkageestimators. However, Butler and Denham (2000) show that it has peculiarproperties, shrinking some of the elements of ˆγ but inflating others.All these various biased estimators have relationships between them. Inparticular, all the present estimators, as well as latent root regression,which is discussed in the next section along with partial least squares,can be viewed as optimizing (˜β − β) ′ X ′ X(˜β − β), subject to differentconstraints for different estimators (see Hocking (1976)). If the data setis augmented by a set of dummy observations, and least squares is usedto estimate β from the augmented data, Hocking (1976) demonstratesfurther that ridge, generalized ridge, PC regression, Marquardt’s modificationand shrinkage estimators all appear as special cases for particular