Chapter 5 Robust Performance Tailoring with Tuning - SSL - MIT

αk. A one-dimensional line search is generally used to find the minimizing stepsize.
In general, this value is not found exactly, instead a termination criteria is used to
determine when the line search algorithm has gotten close enough. Bertsekas suggests
implementations for the line search in [16].
A.2 Newton’s Method
In Newton’s method a quadratic approximation to f around the current iterate xk is
minimized. The descent direction, therefore, includes second-order information:
dk = − � ∇ 2 f (xk) � −1 ∇f (xk) (A.12)
The quantity, ∇ 2 f is the second-order derivate of the objective with respect to the
design variables and is known as the Hessian:
∇ 2 f = ∂2 f (x)
∂x 2
⎡
=
⎢
⎣
∂2f(x) ∂x2 1
∂2f(x) ∂x2x1
.
∂ 2 f(x)
∂xnx1
∂ 2 f(x)
∂x1x2
∂2f(x) ∂x2 2
.
∂ 2 f(x)
∂xnx2
...
...
.. .
...
∂ 2 f(x)
∂x1xn
∂2f(x) ∂x2xn
.
∂ 2 f(x)
∂x 2 n
⎤
⎥
⎦
(A.13)
As with the gradients, the Hessian may be calculated ananlytically or approximated
through finite-difference or other methods. If the central difference equation (Equa-
tion 4.4) is used to calculate gradients, then the diagonal elements of the Hessian can
be obtained at very little additional expesne:
∂ 2 f (x)
∂x 2 i
= f (x +∆x�ei) − f (x − ∆xei) − 2f (x)
∆x 2
(A.14)
Newton’s method is very popular due to its fast convergence properties. In fact,
the method finds the global minimum of a positive definite quadratic function in
only one iteration. For this reason, many modified algorithms are based on Newton’s
method. One drawback of the tecqhnique is that the Hessian is required to determine
222