A Survey of Monte Carlo Tree Search Methods - Department of ...

IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES, VOL. 4, NO. 1, MARCH 2012 26
trees periodically and only on parts of the tree 21 [24].
This is better suited to implementation in a messagepassing
setting, where communication between processes
is limited, e.g. when parallelising across clusters of
machines. Bourki et al. find that slow tree parallelisation
slightly outperforms slow root parallelisation, despite
the increased communication overheads of the former.
The idea of periodically synchronising statistics between
trees is also explored in [91].
6.3.4 UCT-Treesplit
Schaefers and Platzner [192] describe an approach they
call UCT-Treesplit for performing a single MCTS search
efficiently across multiple computer nodes. This allows
an equal distribution of both the work and memory
load among all computational nodes within distributed
memory. Graf et al. [98] demonstrate the application
of UCT-Treesplit in their Go program GOMORRA to
achieve high-level play. GOMORRA scales up to 16 nodes
before diminishing returns reduce the benefit of splitting
further, which they attribute to the high number of
simulations being computed in parallel.
6.3.5 Threading and Synchronisation
Cazenave and Jouandeau [48] describe a parallel Master-
Slave algorithm for MCTS, and demonstrate consistent
improvement with increasing parallelisation until 16
slaves are reached. 22 The performance of their 9 × 9 Go
program increases from 40.5% with one slave to 70.5%
with 16 slaves against GNU GO 3.6.
Enzenberger and Müller [80] describe an approach
to multi-threaded MCTS that requires no locks, despite
each thread working on the same tree. The results
showed that this approach has much better scaling on
multiple threads than a locked approach.
Segal [195] investigates why the parallelisation of
MCTS across multiple machines has proven surprisingly
difficult. He finds that there is an upper bound on the
improvements from additional search in single-threaded
scaling for FUEGO, that parallel speedup depends critically
on how much time is given to each player, and that
MCTS can scale nearly perfectly to at least 64 threads
when combined with virtual loss, but without virtual
loss scaling is limited to just eight threads.
6.4 Considerations for Using Enhancements
MCTS works well in some domains but not in others.
The many enhancements described in this section and
the previous one also have different levels of applicability
to different domains. This section describes efforts to
understand situations in which MCTS and its enhancements
may or may not work, and what conditions might
cause problems.
21. For example, only on nodes above a certain depth or with more
than a certain number of visits.
22. At which point their algorithm is 14 times faster than its sequential
counterpart.
6.4.1 Consistency
Heavily modified MCTS algorithms may lead to incorrect
or undesirable behaviour as computational power
increases. An example of this is a game played between
the Go program MOGO and a human professional,
in which MOGO incorrectly deduced that it was in a
winning position despite its opponent having a winning
killer move, because that move matched a number of
very bad patterns so was not searched once [19]. Modifying
MCTS enhancements to be consistent can avoid
such problems without requiring that the entire search
tree eventually be visited.
6.4.2 Parameterisation of Game Trees
It has been observed that MCTS is successful for tricktaking
card games, but less so for poker-like card games.
Long et al. [130] define three measurable parameters of
game trees and show that these parameters support this
view. These parameters could also feasibly be used to
predict the success of MCTS on new games.
6.4.3 Comparing Enhancements
One issue with MCTS enhancements is how to measure
their performance consistently. Many enhancements lead
to an increase in computational cost which in turn results
in fewer simulations per second; there is often a tradeoff
between using enhancements and performing more
simulations.
Suitable metrics for comparing approaches include:
• Win rate against particular opponents.
• Elo 23 ratings against other opponents.
• Number of iterations per second.
• Amount of memory used by the algorithm.
Note that the metric chosen may depend on the reason
for using a particular enhancement.
7 APPLICATIONS
Chess has traditionally been the focus of most AI games
research and been described as the “drosophila of AI” as
it had – until recently – been the standard yardstick for
testing and comparing new algorithms [224]. The success
of IBM’s DEEP BLUE against grandmaster Gary Kasparov
has led to a paradigm shift away from computer Chess
and towards computer Go. As a domain in which computers
are not yet at the level of top human players, Go
has become the new benchmark for AI in games [123].
The most popular application of MCTS methods is to
games and of these the most popular application is to
Go; however, MCTS methods have broader use beyond
games. This section summarises the main applications
of MCTS methods in the literature, including computer
Go, other games, and non-game domains.
23. A method for calculating relative skill levels between players that
is widely used for Chess and Go, named after Arpad Elo.