Chapter 5 Robust Performance Tailoring with Tuning - SSL - MIT

Appendix A
Gradient-Based Optimization
The goal of a general unconstrained nonlinear programming optimization problem is
to minimize a convex objective function, f, over a set of design variables, x defined
in the set of real numbers, R n :
min f (x) (A.1)
x∈Rn One way to find the optimal set of parameters, x ∗ , is to begin at some initial iterate,
x0 and search the design space by finding successive iterates, xk that reduce the
objective funcion. A general form for the iterate is:
xk+1 = xk + αkdk
(A.2)
where k denotes the iteration number, αk is the stepsize and dk is a serach direction.
It is the choice of the search direction that distinguishes one optimization algorithm
from another. In gradient-based optimizations d is chosen based on the gradient of
the cost function at each iteration. There is a large body of work on this subject
and many algorithms from which to choose. The choice of algorithm depends on the
features of the given problem formulation. Bertsekas provides a detailed overview of
many popular gradient methods in [16]. In this appendix, three popular gradient-
based algorithms are briefly reviewed.
219