Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

34 3. Properties of Sample Principal ComponentsProperty G2 may also be carried over from populations to samples asfollows. Suppose that the observations x 1 , x 2 ,...x n are transformed byy i = B ′ x i ,i =1, 2,...,n,where B is a (p × q) matrix with orthonormal columns, so thaty 1 , y 2 ,...,y n , are projections of x 1 , x 2 ,...,x n onto a q-dimensionalsubspace. Thenn∑h=1 i=1n∑(y h − y i ) ′ (y h − y i )is maximized when B = A q . Conversely, the same criterion is minimizedwhen B = A ∗ q.This property means that if the n observations are projected onto aq-dimensional subspace, then the sum of squared Euclidean distances betweenall pairs of observations in the subspace is maximized when thesubspace is defined by the first q PCs, and minimized when it is definedby the last q PCs. The proof that this property holds is again rather similarto that for the corresponding population property and will not berepeated.The next property to be considered is equivalent to Property A5.Both are concerned, one algebraically and one geometrically, with leastsquares linear regression of each variable x j on the q variables containedin y.Property G3. As before, suppose that the observations x 1 , x 2 ,...,x nare transformed by y i = B ′ x i , i =1, 2,...,n,whereB is a (p × q) matrixwith orthonormal columns, so that y 1 , y 2 ,...,y n are projections ofx 1 , x 2 ,...,x n onto a q-dimensional subspace. A measure of ‘goodness-offit’of this q-dimensional subspace to x 1 , x 2 ,...,x n can be defined as thesum of squared perpendicular distances of x 1 , x 2 ,...,x n from the subspace.This measure is minimized when B = A q .Proof. The vector y i is an orthogonal projection of x i onto a q-dimensional subspace defined by the matrix B. Let m i denote the positionof y i in terms of the original coordinates, and r i = x i − m i . (See Figure3.1 for the special case where p =2,q = 1; in this case y i is a scalar,whose value is the length of m i .) Because m i is an orthogonal projectionof x i onto a q-dimensional subspace, r i is orthogonal to the subspace, sor ′ i m i = 0. Furthermore, r ′ i r i is the squared perpendicular distance of x ifrom the subspace so that the sum of squared perpendicular distances ofx 1 , x 2 ,...,x n from the subspace isn∑r ′ ir i .i=1