C++ for Scientists - Technische Universität Dresden

14.6. THE NEWTON-RAPHSON METHOD FOR FINDING THE MINIMUM OF A CONVEX FUNCTION2
Then p ′ (x) = 0 for
x = ˜x − f ′ (˜x)
f ′′ (˜x)
The Newton method goes as follows : In this algorithm τ is a tolerance for the stopping criterion.
1
2
1. Given initial ˜x = x (0) .
2. Repeat for j = 1, 2, . . .
3
2.1. Compute x (j) = x (j−1) − f ′ (x (j−1) )/f ′′ (x (j−1) 4
)
3. Until |f ′ (x (j−1) )/f ′′ (xj−1) 5
)| < τ
The iteration stops when the derivative is much smaller than the second order derivative. What
happens when f ′′ (xj−1) = 0 ?
14.6.2 Multivariate functions
For multivariate functions, the principle is the same, but it is more complicated. A multivariate
function f has an argument x ∈ R n , ie. a vector of size n. For example, f = sin(x1)+x2 cos(x1)
is a multivariate function in the variables x1 and x2.
We use the same idea as for one variable. That is, we use the Taylor expansion to approximate
the function :
f(x) � f(˜x) + ∇f(˜x) T (x − ˜x) + 1
2 (x − ˜x)T H(f(˜x))(x − ˜x)
⎛ ⎞
∂f/∂x1
⎜ ⎟
∇f(˜x) = ⎝ . ⎠
H(f) =
⎡
⎢
⎣
∂f/∂xn
∂ 2 f/∂x1∂x1 · · · ∂ 2 f/∂x1∂xn
.
∂ 2 f/∂xn∂x1 · · · ∂ 2 f/∂xn∂xn
where ∇f(˜x) is called the gradient vector and H(f) the Hessian matrix.
The derivative becomes
so, the derivative is zero when
f ′ (x) = ∇f(˜x) + H(f(˜x))(x − ˜x)
x = ˜x − {H(f(˜x))} −1 ∇f(˜x) .
This requires the solution of an n × n linear system on each iteration. The Newton algorithm
is very similar to the univariate case:
14.6.3 Software for uni-variate functions
The task is to first develop the function
.
⎤
⎥
⎦