Introduction to Unconstrained Optimization - Scilab

The previous definition intuitively means that h(x) converges faster to zero than
‖x‖ p .
Definition 1.5. (Big o notation) Assume that p is a positive integer and h is a
function h : R n → R. We have
h(x) = O(‖x‖ p ) (20)
if there exists a finite number M > 0, independent of x, and a real number δ > 0
such that
for all ‖x‖ ≤ δ.
|f(x)| ≤ M‖x‖ p , (21)
The equality 20 therefore implies that the rate at which h(x) converges to zero
increases as p increases.
1.8 Various Taylor expansions
In this section, we will review various results which can be applied to continuously
differentiable functions. These results are various forms of Taylor expansions which
will be used throughout this document. We will not prove these propositions, which
are topic of a general calculus course.
The following proposition makes use of the gradient of the function f.
Proposition 1.6. (Mean value theorem) Let f : R n → R be continously differentiable
on an open set S and let x ∈ S. Therefore, for all p such that x + p ∈ S,
there exists an α ∈ [0, 1] such that
f(x + p) = f(x) + p T g(x + αp). (22)
The following proposition makes use of the Hessian matrix of the function f.
Proposition 1.7. (Second order expansion) Let f : R n → R be twice continously
differentiable on an open set S and let x ∈ S. Therefore, for all p such that x + p ∈
S, there exists an α ∈ [0, 1] such that
f(x + p) = f(x) + p T g(x) + 1 2 pT H(x + αp)p. (23)
There is an alternative form of the second order expansion, which makes use of
the o(·) or O(·) notations.
Proposition 1.8. (Second order expansion - second form) Let f : R n → R be twice
continously differentiable on an open set S and let x ∈ S. Therefore, for all p such
that x + p ∈ S
f(x + p) = f(x) + p T g(x) + 1 2 pT H(x)p + O(‖p‖ 3 ). (24)
The last term in the previous equation is often written as o(‖p‖ 2 ), which is
equivalent.
15