The Steepest Descent Algorithm for Unconstrained Optimization and ...

A natural consequence of this is the following algorithm, called the steepest
descent algorithm.
Steepest Descent Algorithm:
Step 0. Given x 0 ,set k := 0
Step 1. d k := −∇f (x k ). If d k = 0, then stop.
Step 2. Solve min α f (x k + αd k ) for the stepsize α k , perhaps chosen by
an exact or inexact linesearch.
Step 3. Set x k+1 ← x k + α k d k ,k ← k +1. Go to Step 1.
Note from Step 2 and the fact that d k = −∇f (x k ) is a descent direction,
it follows that f (x k+1 ) < f (x k ).
2 Global Convergence
We have the following theorem:
Convergence Theorem: Suppose that f (·) : R n →R is continuously
differentiable on the set S = {x ∈ R n | f (x) ≤ f (x 0 )}, and that S is a closed
and bounded set. Then every point ¯x that is a cluster point of the sequence
{x k } satisfies ∇f (¯x) =0.
Proof: The proof of this theorem is by contradiction. By the Weierstrass
Theorem, at least one cluster point of the sequence {x
k
} must exist. Let x ¯ be
any such cluster point. Without loss of generality, assume that lim
k k→∞ x =
x, ¯ but that ∇f (¯ x) = ̸ 0. This being the case, there is a value of α> ¯ 0
such that δ := f (¯ x) − f (¯ x + αd) ¯¯ > 0, where d ¯ = −∇f (¯ x). Then also
(¯ x + αd) ¯¯ ∈ intS.
3