A Survey of Monte Carlo Tree Search Methods - Department of ...

IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES, VOL. 4, NO. 1, MARCH 2012 24
Fig. 6. Patterns for a cut move in Go [96].
simulations by detecting pattern matches with the actual
board position and applying associated moves.
For example, Figure 6 from [96] shows a set of 3 × 3
patterns for detecting cut moves in Go. The first pattern
must be matched and the other two not matched for the
move to be recognised. Drake and Uurtamo [77] suggest
there may be more to gain from applying heuristics such
as patterns to the simulation policy rather than the tree
policy for Go.
The 3 × 3 patterns described by Wang and Gelly
[228] vary in complexity, and can be used to improve
the simulation policy to make simulated games more
realistic. Wang and Gelly matched patterns around the
last move played to improve the playing strength of
their Go program MOGO [228]. Gelly and Silver [92]
used a reinforcement learning approach to improve the
simulation policy, specifically a function approximator
QRLGO(s, a), which applied linear weights to a collection
of binary features. 20 Several policies using this information
were tested and all offered an improvement over
a random policy, although a weaker handcrafted policy
was stronger when used with UCT.
Coulom [71] searched for useful patterns in Go by
computing Elo ratings for patterns, improving their Go
program CRAZY STONE. Hoock and Teytaud investigate
the use of Bandit-based Genetic Programming (BGP) to
automatically find good patterns that should be more
simulated and bad patterns that should be less simulated
for their program MOGO, achieving success with 9 × 9
Go but less so with 19 × 19 Go [105].
Figure 7 shows a bridge pattern that occurs in connection
games such as Hex, Y and Havannah (7.2). The
two black pieces are virtually connected as an intrusion
by white in either cell can be answered by black at
the other cell to restore the connection. Such intrusions
can be detected and completed during simulation to
significantly improve playing strength, as this mimics
moves that human players would typically perform.
6.2 Backpropagation Enhancements
Modifications to the backpropagation step typically involve
special node updates required by other enhancement
methods for forward planning, but some constitute
enhancements in their own right. We describe those not
explicitly covered in previous sections.
20. These features were all 1 × 1 to 3 × 3 patterns on a Go board.
a b
Fig. 7. Bridge completion for connection games.
6.2.1 Weighting Simulation Results
b a
Xie and Liu [237] observe that some simulations are
more important than others. In particular, simulations
performed later in the search tend to be more accurate
than those performed earlier, and shorter simulations
tend to be more accurate than longer ones. In light of
this, Xie and Liu propose the introduction of a weighting
factor when backpropagating simulation results [237].
Simulations are divided into segments and each assigned
a positive integer weight. A simulation with weight w is
backpropagated as if it were w simulations.
6.2.2 Score Bonus
In a normal implementation of UCT, the values backpropagated
are in the interval [0, 1], and if the scheme
only uses 0 for a loss and 1 for a win, then there is no
way to distinguish between strong wins and weak wins.
One way of introducing this is to backpropagate a value
in the interval [0, γ] for a loss and [γ, 1] for a win with
the strongest win scoring 1 and the weakest win scoring
γ. This scheme was tested for Sums Of Switches (7.3) but
did not improve playing strength [219].
6.2.3 Decaying Reward
Decaying reward is a modification to the backpropagation
process in which the reward value is multiplied by
some constant 0 < γ ≤ 1 between each node in order
to weight early wins more heavily than later wins. This
was proposed alongside UCT in [119], [120].
6.2.4 Transposition Table Updates
Childs et al. [63] discuss a variety of strategies – labelled
UCT1, UCT2 and UCT3 – for handling transpositions
(5.2.4), so that information can be shared between
different nodes corresponding to the same state. Each
variation showed improvements over its predecessors,
although the computational cost of UCT3 was large.
6.3 Parallelisation
The independent nature of each simulation in MCTS
means that the algorithm is a good target for parallelisation.
Parallelisation has the advantage that more
simulations can be performed in a given amount of time
and the wide availability of multi-core processors can
be exploited. However, parallelisation raises issues such
as the combination of results from different sources in a
single search tree, and the synchronisation of threads of