The Steepest Descent Algorithm for Unconstrained Optimization and ...

∗
the optimal solution x = x . In order to see how this arises, we will examine
the case where the objective function f (x) is itself a simple quadratic
function of the form:
f (x) = 1 x T Qx + q T x,
2
where Q is a positive definite symmetric matrix. We will suppose that the
eigenvalues of Q are
A = a 1 ≥ a 2 ≥ ... ≥ a n = a > 0,
i.e, A and a are the largest and smallest eigenvalues of Q.
The optimal solution of P is easily computed as:
∗
x = −Q −1 q
and direct substitution shows that the optimal objective function value is:
1 T
Q −1
f (x ∗ )= − 2
q q.
For convenience, let x denote the current point in the steepest descent
algorithm. We have:
f (x) = 1 x T Qx + q T x
2
and let d denote the current direction, which is the negative of the gradient,
i.e.,
d = −∇f (x) = −Qx − q.
Now let us compute the next iterate of the steepest descent algorithm.
If α is the generic step-length, then
1
f (x + αd) = 2
(x + αd) T Q(x + αd)+ q T (x + αd)
5