Introduction to Unconstrained Optimization - Scilab

1 Overview
In this section, we analyze optimization problems and define the associated vocabulary.
We introduce level sets and separate local and global optimums. We emphasize
the use of contour plots in the context of unconstrained and constrained optimization.
1.1 Classification of optimization problems
In this document, we consider optimization problems in which we try to minimize a
cost function
min f(x) (1)
x∈Rn with or without constraints. Several properties of the problem to solve may be taken
into account by the numerical algorithms:
• The unknown may be a vector of real or integer values.
• The number of unknowns may be small (from 1 to 10 - 100), medium (from
10 to 100 - 1 000) or large (from 1 000 - 10 000 and above), leading to dense
or sparse linear systems.
• There may be one or several cost functions (multi-objective optimization).
• The cost function may be smooth or non-smooth.
• There may be constraints or no constraints.
• The constraints may be bounds constraints, linear or non-linear constraints.
• The cost function can be linear, quadratic or a general non linear function.
An overview of optimization problems is presented in figure 1. In this document,
we will be concerned mainly with continuous parameters and problems with one
objective function only. From that point, smooth and nonsmooth problems require
very different numerical methods. It is generally believed that equality constrained
optimization problems are easier to solve than inequality constrainted problems.
This is because computing the set of active constraints at optimum is a difficult
problem.
The size of the problem, i.e. the number n of parameters, is also of great concern
with respect to the design of optimization algorithms. Obviously, the rate of convergence
is of primary interest when the number of parameters is large. Algorithms
which require n iterations, like BFGS’s methods for example, will not be efficient
if the number of iterations to perform is much less than n. Moreover, several algorithms
(like Newton’s method for example), require too much storage to be of
a practical value when n is so large that n × n matrices cannot be stored. Fortunately,
good algorithms such as conjugate gradient methods and BFGS with limited
memory methods are specifically designed for this purpose.
4