Summary for simple linear regression Dependent data x Linear ...

Summary for simple linear regression
Dependent data x i , y i
Linear model
ˆy i = b 0 + b 1 x i i = 1,...,n
n
n( ) = ! ( ) ; " =
i
i=1
b0 = y ! b x ; 1
n
x' y'
x' 2 y' 2
b 1 =
i = 1,...,n
x' y'
x' 2 = ! sy sx y i = ˆy i + ! i
SST = y ' i 2
! = ny ' i 2 is the total variance of y (with n-1 d.o.f.)
i=1
n
2
SSE = " ! = SST 1# $ i
i=1
2
( ) is the residual (error) variance, with n-2 d.o.f, or in
the case of multiple regression, with K predictors, with n-K-1 d.o.f.
SSR = SST ! SSE = SST "2 is the “explained variance”, with 1 d.o.f. (or K d.o.f.
in the case of multiple regression).
Generalized coefficient of determination R 2 :
R 2 = SSR
SST
= SST ! SSE
SST
dependent sample)
=
y ' i 2
n
n
2
" ! " # i
i=1 i=1
y ' i 2
n
Estimation of errors in the regression coefficients:
"
i=1
: “explained variance” (for the
The coefficient b1, divided by the standard deviation obtained from the
sample has a tn-2 random distribution:
24

! E(b )
1 1
" SSE %
#
$ n ! 2&
' / x' ~ t
n
n!2
2
( i
i=1
This means that we can test whether b1 is significantly different from zero by
using the t table 2 with n-2 d.o.f. at a level of significance of, let’s say 5%.
The 95% confidence interval for b1 is
)
+ b ! 1
* +
Similarly
" SSE %
#
$ n ! 2&
' / x' n
2
i
b 0 ! E(b 0 )
" SSE % 1
#
$ n ! 2&
' *
n
determined.
n
(
i=1
( * t ,b +
.025,n!2 1
i=1
2
x'i " SSE %
#
$ n ! 2&
' / x' n
2
i
( * t .025,n!2
i=1
~ t n!2 and the limits of confidence for b0 are similarly
For a new predictor x0, the limits of confidence of the new prediction can be
obtained from the fact that it has a distribution
y(x 0 ) ! b 0 ! b 1 x 0
" SSE % 1
#
$ n ! 2&
' * 1+
n + x "
$ ! x 0
$
n
$
(
#
$
( ) 2
i=1
2
x'i ~ tn!2 %
'
'
'
&
'
We will see more about this after the multiple regression discussion.
Note that a “naïve” estimation of the forecast error variance
! 2 = 1
n
2
" ! # i
n
SSE
seriously underestimates the error even for the dependent
n
i=1
2 2 1
sample, for which the correct formula is s = ! = !
n
,
.
-.
n
2
" ! = i
i=1
SSE
n # K # 1 .
25

Analysis of residuals: Ideally the residuals (forecast errors) should look
random, without trends. If we find, for example
y
x
y-b0-b1x
If there is a trend, it is better to change variables, for the predictor,
e.g., y = b 0 + b 1 f (x), y = b 0 + b 1 x + b 2 x2 + ... + b K x K . Note that this is still a linear
regression, with multiple predictors, even if the predictors are nonlinear
functions of x.
0
x
26