Introduction to Unconstrained Optimization - Scilab

x 2
x*
x 1
Figure 6: Unconstrained optimization problem with positive definite Hessian
1.6 An optimum in unconstrained optimization
Before getting into the mathematics, we present some intuitive results about unconstrained
optimization.
Suppose that we want to solve the following unconstrained optimization problem.
min f(x) (17)
x∈Rn where f is a smooth objective function. We suppose here that the hessian matrix is
positive definite, i.e. its eigenvalues are strictly positive. This optimization problem
is presented in figure 6, where the contours of the objective function are drawn.
The contours are the locations where the objective has a constant value. When
the function is smooth and if we consider the behaviour of the function very near the
optimum, the contours are made of ellipsoids : when the ellipsoid is more elongated,
the eigenvalues are of very different magnitude. This behaviour is the consequence of
the fact that the objective function can be closely approximated, near the optimum,
by a quadratic function, as expected by the local Taylor expansion of the function.
This quadratic function is closely associated with the Hessian matrix of the objective
function.
1.7 Big and little O notations
The proof of several results associated with optimality require to make use of Taylor
expansions. Since the development of these expansions may require to use the big
and little O notations, this is the good place to remind ourselves the definition of
the big and little o notations.
Definition 1.4. (Little o notation) Assume that p is a positive integer and h is a
function h : R n → R.
We have
h(x) = o(‖x‖ p ) (18)
if
lim
x→0
h(x)
= 0. (19)
‖x‖
p
14