What Is Optimization Toolbox?

3 Standard AlgorithmsUnconstrained Optimization• “Introduction” on page 3-6• “Quasi-Newton Methods” on page 3-7• “Line Search” on page 3-9IntroductionAlthough a wide spectrum of methods exists for unconstrained optimization,methods can be broadly categorized in terms of the derivative information thatis, or is not, used. Search methods that use only function evaluations (e.g., thesimplex search of Nelder and Mead [32]) are most suitable for problems thatare very nonlinear or have a number of discontinuities. Gradient methods aregenerally more efficient when the function to be minimized is continuous inits first derivative. Higher order methods, such as Newton’s method, areonly really suitable when the second-order information is readily and easilycalculated, because calculation of second-order information, using numericaldifferentiation, is computationally expensive.Gradient methods use information about the slope of the function to dictatea direction of search where the minimum is thought to lie. The simplest ofthese is the method of steepest descent in which a search is performed in adirection, ,where is the gradient of the objective function. Thismethod is very inefficient when the function to be minimized has long narrowvalleys as, for example, is the case for Rosenbrock’s function(3-2)The minimum of this function is at where .Acontourmap of this function is shown in the figure below, along with the solutionpath to the minimum for a steepest descent implementation starting at thepoint [-1.9,2]. The optimization was terminated after 1000 iterations, still aconsiderable distance from the minimum. The black areas are where themethod is continually zigzagging from one side of the valley to another. Notethat toward the center of the plot, a number of larger steps are taken when apoint lands exactly at the center of the valley.3-6