GLS (Generalized Least Squares) - Paul Johnson Homepage

In matrix form, the representation of WLS/GLS is
S(ˆb) = (y − ŷ) ′ W (y − ŷ) (12)
and, if you write that out, the sum of squares in a GLS framework is a rather more complicated
thing. All of those off-diagonal w ij ’s make the number of terms multiply.
⎡
⎤ ⎡
⎤
w 11 w 12 · · · w 1N y 1 − ŷ 1
[ ]
w 12 w 22 w 2N
y 2 − ŷ 2
y1 − ŷ 1 y 2 − ŷ 2 · · · y N − ŷ N ⎢
⎣ .
.. .
⎥ ⎢
⎥
. ⎦ ⎣ . ⎦
w N1 w N2 · · · w NN y N − ŷ N
= [ ]
y 1 − ŷ 1 y 2 − ŷ 2 · · · y N − ŷ N ⎢
⎣
⎡
w 11 (y 1 − ŷ 1 ) +w 12 (y 2 − ŷ 2 ) · · · +w 1N (y N − ŷ N )
w 12 (y 1 − ŷ 1 ) +w 22 (y 2 − ŷ 2 ) +w 2N (y N − ŷ N )
.
.
.. .
w N1 (y 1 − ŷ 1 ) +w N2 (y 2 − ŷ 2 ) · · · +w NN (y N − ŷ N )
⎤
⎥
⎦
⎡
=
⎢
⎣
w 11 (y 1 − ŷ 1 ) 2 ⎤
+w 12 (y 2 − ŷ 2 )(y 1 − ŷ 1 ) · · · +w 1N (y N − ŷ N )(y 1 − ŷ 1 )
w 12 (y 1 − ŷ 1 )(y 2 − ŷ 2 ) +w 22 (y 2 − ŷ 2 ) 2 +w 2N (y N − ŷ N )(y 2 − ŷ 2 )
.
.. .
⎥
.
⎦
w N1 (y 1 − ŷ 1 )(y N − ŷ N ) +w N2 (y 2 − ŷ 2 )(y N − ŷ N ) · · · +w NN (y N − ŷ N ) 2
w 11 (y 1 − ŷ 1 ) 2 + w 12 (y 2 − ŷ 2 )(y 1 − ŷ 1 ) + · · · + w 1N (y N − ŷ N )(y 1 − ŷ 1 )
= +w 12 (y 1 − ŷ 1 )(y 2 − ŷ 2 ) + w 22 (y 2 − ŷ 2 ) 2 + · · · + w 2N (y N − ŷ N )(y 2 − ŷ 2 )
+w N1 (y 1 − ŷ 1 )(y N − ŷ N ) + w N2 (y 2 − ŷ 2 )(y N − ŷ N ) + · · · + w NN (y N − ŷ N ) 2
Supposing a linear model:
N∑ N∑
= w ij (y i − ŷ i )(y j − ŷ j ) (13)
i=1 j=1
ŷ = Xˆb (14)
The normal equations (the name for the first order conditions) are found by setting the derivatives
of the Sum of Squares with respect to the parameters equal to 0.
for each parameter ˆb j :
∂
∂ˆb j
[(y − ŷ) ′ W (y − ŷ)] = 0
If you insert ŷ = Xb in there and do the math (or look it up in a book :) )
(X ′ W X)ˆb − X ′ W y = 0
The WLS/GLS estimator is the solution:
ˆb = (X ′ W X) −1 X ′ W y (15)
3