Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

222 9. Principal Components Used with Other Multivariate Techniquesparameters in the π g , µ g and Σ g is 1 2 G[p2 +3p +2]− 1. To overcome thisintractability, the elements of the Σ g can be constrained in some way, anda promising approach was suggested by Tipping and Bishop (1999b), basedon their probabilistic model for PCA which is described in Section 3.9. Inthis approach, the p.d.f.s in (9.2.2) are replaced by p.d.f.s derived from Tippingand Bishop’s (1999a,b) model. These p.d.f.s are multivariate normal,but instead of having general covariance matrices, the matrices take theform B g B ′ g + σ 2 gI p , where B g is a (p × q) matrix, and q (< p) is the samefor all groups. This places constraints on the covariance matrices, but theconstraints are not as restrictive as the more usual ones, such as equality ordiagonality of matrices. Tipping and Bishop (1999b) describe a two-stageEM algorithm for finding maximum likelihood estimates of all the parametersin the model. As with Tipping and Bishop’s (1999a) single populationmodel, it turns out the columns of B g define the space spanned by the firstq PCs, this time within each cluster. There remains the question of thechoice of q, and there is still a restriction to multivariate normal distributionsfor each cluster, but Tipping and Bishop (1999b) provide exampleswhere the procedure gives clear improvements compared to the impositionof more standard constraints on the Σ g . Bishop (1999) outlines how aBayesian version of Tipping and Bishop’s (1999a) model can be extendedto mixtures of distributions.9.3 Canonical Correlation Analysis and RelatedTechniquesCanonical correlation analysis (CCA) is the central topic in this section.Here the variables are in two groups, and relationships are sought betweenthese groups of variables. CCA is probably the most commonly used techniquefor tackling this objective. The emphasis in this section, as elsewherein the book, is on how PCA can be used with, or related to, the technique.A number of other methods have been suggested for investigatingrelationships between groups of variables. After describing CCA, and illustratingit with an example, some of these alternative approaches are brieflydescribed, with their connections to PCA again highlighted. Discussion oftechniques which analyse more than two sets of variables simultaneously islargely deferred to Section 14.5.9.3.1 Canonical Correlation AnalysisSuppose that x p1 , x p2 are vectors of random variables with p 1 , p 2 elements,respectively. The objective of canonical correlation analysis (CCA) is to findsuccessively for k =1, 2,..., min[p 1 ,p 2 ], pairs {a ′ k1 x p 1, a ′ k2 x p 2} of linearfunctions of x p1 , x p2 , respectively, called canonical variates, such that the