Prime Numbers

5.2 Monte Carlo methods 231
It is certainly possible for the gcd at Step [Bad seed] to be n itself, and the
chance for this is enhanced if one uses the above idea to put off performing
gcd’s. However, this defect can be mitigated by storing the values U, V at the
last gcd. If the next gcd is n, one can return to the stored values U, V and
proceed one step at a time, performing a gcd at each step.
There are actually many choices for the function F (x). The key criterion is
that the iterates of F modulo p should not have long ρ’s, or as [Guy 1976] calls
them, “epacts.” The epact of a prime p with respect to a function F from Zp to
Zp is the largest k for which there is an s with F (0) (s),F (1) (s),...,F (k) (s) all
distinct. (Actually we have taken some liberty with this definition, originally
Guy defined it as the number of iterates to discover the factor p.)
So a poor choice for a function F (x) isax + b, since the epact for a prime
p is the multiplicative order of a modulo p (when a ≡ 1(modp)), usually a
large divisor of p − 1. (When a ≡ 1(modp) andb ≡ 0(modp), the epact is
p.)
Even among quadratic functions x 2 + b there can be poor choices, for
example b = 0. Another less evident, but nevertheless poor, choice is x2 −2. If
x can be represented as y + y−1 modulo p, then the k-th iterate is y2k + y−2k modulo p.
It is not known whether the epact of x2 +1 forp is a suitably slow-growing
function of p, but Guy conjectures it is O √ p ln p .
If we happen to know some information about the prime factors p of n, it
may pay to use higher-degree polynomials. For example, since all prime factors
of the Fermat number Fk are congruent to 1 (mod 2k+2 )whenk≥2 (see
Theorem 1.3.5), one might use x2k+2 + 1 for the function F when attempting
to factor Fk by the Pollard rho method. One might expect the epact for
a prime factor p of Fk to be smaller than that of x2 + 1 by a factor of
about √ 2k+1 . To see this consider the following probabilistic model. (Note
that a more refined probabilistic model that agrees somewhat better with the
available data is given in [Brent and Pollard 1981]. Also see Exercise 5.2.)
Iterating x2 + 1 might be thought of as a random walk through the set of
squares plus 1, a set of size (p − 1)/2, while using x2k+2 + 1 we walk through
the 2k+2 powers plus 1, a set of size (p − 1)/2k+2 . The birthday paradox says
we should expect a repeat in about c √ m steps in a random walk through a
set of size m, so we see the improved factor of √ 2k+1 . However, there is a
penalty to using x2k+2 + 1, since a typical loop now involves 3(k + 2) modular
squarings and one modular multiplication. For large k the benefit is evident.
In this connection see Exercise 5.24. Such acceleration was used successfully
in [Brent and Pollard 1981] to factor F8, historically the most spectacular
factorization achieved with the Pollard rho method. The work of Brent and
Pollard also discusses a somewhat faster cycle-finding method, which is to save
certain iterate values and comparing future ones with those, as an alternative
to the Floyd cycle-finding method.