The Steepest Descent Algorithm for Unconstrained Optimization and ...

1 The Algorithm
The problem we are interested in solving is:
P : minimize f(x)
s.t. x ∈R n ,
where f(x) is differentiable. If x =¯x is a given point, f(x) can be approximated
by its linear expansion
f(¯ x + d) ≈ f(¯ x)+ ∇f(¯ x) T d
if d “small”, i.e., if ‖d‖ is small. Now notice that if the approximation in
the above expression is good, then we want to choose d so that the inner
product ∇f(¯x) T d is as small as possible. Let us normalize d so that ‖d‖ = 1.
Then among all directions d with norm ‖d‖ = 1, the direction
d˜= −∇f(¯x)
‖∇f(¯x)‖
makes the smallest inner product with the gradient ∇f(¯x). This fact follows
from the following inequalities:
( )
∇f(¯ x)‖‖d‖ = ∇f(¯) T −∇f(¯
x) T x)
d ≥ −‖∇f(¯ x
= −∇f(¯) x T d. ˜
‖∇f(¯ x)‖
For this reason the un-normalized direction:
d¯= −∇f(¯x)
is called the direction of steepest descent at the point ¯x.
Note that d¯ = −∇f(¯ x) is a descent direction as long as ∇f(¯ x) ̸=0. To
see this, simply observe that d¯T ∇f(¯ x) = −(∇f(¯ x)) T ∇f(¯ x) < 0solongas
∇f(¯ x) ̸=0.
2