The Steepest Descent Algorithm for Unconstrained Optimization and ...

A
a
Upper Bound on
δ =
( A/a−1
A/a+1
) 2
Number of Iterations to Reduce
the Optimality Gap by a factor of 10
1.1 1.0 0.0023 1
3.0 1.0 0.25 2
10.0 1.0 0.67 6
100.0 1.0 0.96 58
200.0 1.0 0.98 116
400.0 1.0 0.99 231
Table 1: Sensitivity of Steepest Descent Convergence Rate to the Eigenvalue
Ratio
andinstep2we need to solve α k =arg min α h(α) = f (x k + αd k ). In this
example we will be able to derive an analytic expression for α k . Notice that
h(α) = f (x k + αd k )
=
k k k k
5(x 1 + αd k 1) 2 +(x 2) 2 +4(x 1 + αd k 2 + αd k 1)(x 2 + αd2) k −
k
k
−14(x 1) − 6(x 2 + αd k
1 + αd k 2)+20,
and this is a simple quadratic function of the scalar α. It is minimized at
α k =
(d k 1 )2 +(d2 k )2
2(5(d k 2 )2 +4d k 2 )
1 )2 +(d k 1 dk
Using the steepest descent algorithm to minimize f (x) starting from
1 1
x 1 =(x 1 ,x 2 )=(0, 10), and using a tolerance of ɛ = 10 −6 , we compute the
iterates shown in Table 2 and in Figure 2:
For a convex quadratic function f (x) = 1 x T Qx−c T x, the contours of the
2
function values will be shaped like ellipsoids, and the gradient vector ∇f (x)
at any point x will be perpendicular to the contour line passing through x,
see Figure 1.
8