Chapter 5 Robust Performance Tailoring with Tuning - SSL - MIT

such that the gradient direction is almost orthogonal to the direction leading to the
minimum.
A.1.1 Stepsize Selection
There many possible choices for the stepsize, αk. Perhaps one of the simplest is the
constant stepsize, in which a fixed stepsize, s>0, is selected for all iterates:
αk = s, k =0, 1,... (A.7)
Although easy to implement, a constant stepsize can cause the algorithm to diverge
if s is too large or result in very slow convergence if s is too small.
Another relatively simple rule is the diminishing stepsize in which αk converges
to zero with each successive iteration:
αk → 0 (A.8)
This rule does not guarantee descent and it is possible that the stepsize may become
so small that progress is effectively halted before an optimum is reached. To prevent
this from happening the following condition is imposed:
∞�
αk = ∞ (A.9)
k=0
αk = α0
√k
(A.10)
Equation A.10 is a possible choice for a decreasing stepsize rule. Convergence with
this rule is also known to be slow.
A more complicated but better-behaved method is the limited minimization rule:
f (xk + αkdk) = min
α∈[0,s] f (xk + αdk) (A.11)
In this method, αk is chosen such that the cost function is minimized along the
direction dk. A fixed positive scalar, s, may be chosen to limit the possible size of
221