What Is Optimization Toolbox?

Quasi-Newton ImplementationQuasi-Newton ImplementationThis section describes the implementation of the quasi-Newton method in thetoolbox. The algorithm consists of two phases:• “Hessian Update” on page 3-11• “Line Search Procedures” on page 3-11Hessian UpdateMany of the optimization functions determine the direction of search byupdating the Hessian matrix at each iteration, using the BFGS method(Equation 3-6). The function fminunc also provides an option to use theDFP method given in “Quasi-Newton Methods” on page 3-7 (set HessUpdateto 'dfp' in options to select the DFP method). The Hessian, H, isalwaysmaintained to be positive definite so that the direction of search, d, isalwaysin a descent direction. This means that for some arbitrarily small step inthe direction d, the objective function decreases in magnitude. You achievepositive definiteness of H by ensuring that H is initialized to be positivedefinite and thereafter (from Equation 3-11) is always positive. Theterm is a product of the line search step length parameter andacombination of the search direction d with past and present gradientevaluations,(3-11)You always achieve the condition that is positive by performing asufficiently accurate line search. This is because the search direction, d, isadescent direction, so that and negative gradient of are alwayspositive. Thus, the possible negative termcan be made as smallin magnitude as required by increasing the accuracy of the line search.Line Search ProceduresAfter choosing the direction of the search, the optimization function uses aline search procedure to determine how far to move in the search direction.This section describes the line search procedures used by the functionslsqnonlin, lsqcurvefit, andfsolve.3-11