Multivariate Calculus - Bruce E. Shapiro

110 LECTURE 14. UNCONSTRAINED OPTIMIZATION
Least Squares Linear Regression
Suppose we have a large set of data points in the xy-plane
{(x i , y i ) : i = 1, 2, ..., n}
and we want to find the “best fit” straight line to our data, namely, we want to find
number m and b such that
y = mx + b
is the “best” possible line in the sense that it minimizes the total sum-squared
vertical distance between the data points and the line.
Figure 14.3: The least squares procedure finds the line that minimizes the total sum
of all the vertical distances as shown between the line and the data points.
The vertical distance between any point (x i , y i ) and the line, which we will denote
by d i , is
d i = |mx i + b − y i |
Since this distance is also minimized when its square is minimized, we instead calculate
d 2 i = (mx i + b − y i ) 2
The total of all these square-distances (the “sum-squared-distance”) is
f(m, b) =
n∑
d 2 i =
i=1
n∑
(mx i + b − y i ) 2
i=1
The only unknowns in this expression are the slope m and y-intercept b. Thus we
have written the expression as a function f(m, b). Our goal is to find the values of
m and b that correspond to the global minimum of f(m, b).
Revised December 6, 2006. Math 250, Fall 2006