MACHINE LEARNING TECHNIQUES - LASA

193
9.4 Estimators
9.4.1 Gradient descent
(from Mathworld)
Gradient descent is an incremental hill-climbing algorithm that approaches a minimum or
maximum of a function by taking steps proportional to the gradient at the current point.
An algorithm for finding the nearest local minimum of a function, which presupposes that the
gradient of the function can be computed. The method of steepest descent, also called the
gradient descent method, starts at a point P
i +
and, as many times as needed, moves from P
1
i
to
−∇ the local downhill gradient.
P +
by minimizing along the line extending from ( )
i 1
f P i
When applied to a 1-dimensional function f(x), the method takes the form of iterating
( )
x = x − ε f x
(8.37)
i i−1 i−1
from a starting point
0
x for small ε > 0
above for the functions ( )
3 2
until a fixed point is reached. The results are illustrated
f x = x − 2x
+ 2 with ε = 0.1and starting point x
0
= 2 and 0.01,
respectively.
This method has the severe drawback of requiring a great many iterations for functions, which
have long, narrow valley structures. In such cases, a conjugate gradient method is preferable.
© A.G.Billard 2004 – Last Update March 2011