Chapter 5 Robust Performance Tailoring with Tuning - SSL - MIT

the descent direction. In some cases, calculation of the Hessian may be impossible
or require prohibitively expensive computation. In addition, if the Hessian matrix is
ill-conditioned then the optimization may have trouble converging.
A.3 Conjugate Gradient
Conjugate direction methods were originally developed for solving quadratic problems
and were motivated by a desire to speed up the convergence of steepest descent while
avoiding the exta storage and computation necessary for Newton’s method. The
algorithm can also be applied to the general non-quadratic problem (Equation A.1)
and proceeds with successive iterations as in Equation A.2. The stepsize, αk, must
be obtained through line minimization ((A.11)), and the search directions for this
method are functions of the gradient at the current iteration as well as the previous
search direction:
dk = −∇f (xk)) + βkdk−1
(A.15)
There are multiple forms of the quantity βk. One form, known as the Fletcher-Reeves
conjugate direction [45], requires a quadratic objective function f(x) and a perfect
line search for the stepsize. Another common form is the Polak-Ribiere conjugate di-
rection which relaxes the quadratic requirement on f(x), but only prodcues conjugate
directions if the line search is accurate. A more general direction, requiring neither a
quadratic objective nor a perfect line search, is proposed by Perry[101]:
βk = (∇f (xk) −∇f (xk−1) − αk−1dk−1) T ∇f (xk)
(∇f (xk) −∇f (xk−1)) T dk−1
(A.16)
The conjugate gradient method is essentially a compromise between the simplicity
of steepest descent and fast convergence of Newton’s method. When there are non-
quadratic terms in the cost function there is the danger for loss of conjugancy as
the algorithm progresses. Therefore it is common practice to periodically restart the
algorithm with a steepest descent direction.
223