A Survey of Monte Carlo Tree Search Methods - Department of ...

IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES, VOL. 4, NO. 1, MARCH 2012 14
4.8.7 Monte Carlo α-β (MCαβ)
Monte Carlo α-β (MCαβ) combines MCTS with traditional
tree search by replacing the default policy
with a shallow α-β search. For example, Winands and
Björnnsson [232] apply a selective two-ply α-β search in
lieu of a default policy for their program MCαβ, which
is currently the strongest known computer player for the
game Lines of Action (7.2). An obvious consideration in
choosing MCαβ for a domain is that a reliable heuristic
function must be known in order to drive the α-β
component of the search, which ties the implementation
closely with the domain.
4.8.8 Monte Carlo Counterfactual Regret (MCCFR)
Counterfactual regret (CFR) is an algorithm for computing
approximate Nash equilibria for games of imperfect
information. Specifically, at time t + 1 the policy plays
actions with probability proportional to their positive
counterfactual regret at time t, or with uniform probability
if no actions have positive counterfactual regret.
A simple example of how CFR operates is given in [177].
CFR is impractical for large games, as it requires
traversal of the entire game tree. Lanctot et al. [125]
propose a modification called Monte Carlo counterfactual
regret (MCCFR). MCCFR works by sampling blocks of
terminal histories (paths through the game tree from root
to leaf), and computing immediate counterfactual regrets
over those blocks. MCCFR can be used to minimise,
with high probability, the overall regret in the same
way as CFR. CFR has also been used to create agents
capable of exploiting the non-Nash strategies used by
UCT agents [196].
4.8.9 Inference and Opponent Modelling
In a game of imperfect information, it is often possible
to infer hidden information from opponent actions, such
as learning opponent policies directly using Bayesian
inference and relational probability tree learning. The
opponent model has two parts – a prior model of
a general opponent, and a corrective function for the
specific opponent – which are learnt from samples of
previously played games. Ponsen et al. [159] integrate
this scheme with MCTS to infer probabilities for hidden
cards, which in turn are used to determinize the cards for
each MCTS iteration. When the MCTS selection phase
reaches an opponent decision node, it uses the mixed
policy induced by the opponent model instead of banditbased
selection.
4.8.10 Simultaneous Moves
Simultaneous moves can be considered a special case
of hidden information: one player chooses a move but
conceals it, then the other player chooses a move and
both are revealed.
Shafiei et al. [196] describe a simple variant of UCT
for games with simultaneous moves. Shafiei et al. [196]
argue that this method will not converge to the Nash
equilibrium in general, and show that a UCT player can
thus be exploited.
Teytaud and Flory [216] use a similar technique to
Shafiei et al. [196], the main difference being that they
use the EXP3 algorithm (5.1.3) for selection at simultaneous
move nodes (UCB is still used at other nodes). EXP3
is explicitly probabilistic, so the tree policy at simultaneous
move nodes is mixed. Teytaud and Flory [216]
find that coupling EXP3 with UCT in this way performs
much better than simply using the UCB formula at
simultaneous move nodes, although performance of the
latter does improve if random exploration with fixed
probability is introduced.
Samothrakis et al. [184] apply UCT to the simultaneous
move game Tron (7.6). However, they simply avoid
the simultaneous aspect by transforming the game into
one of alternating moves. The nature of the game is
such that this simplification is not usually detrimental,
although Den Teuling [74] identifies such a degenerate
situation and suggests a game-specific modification to
UCT to handle this.
4.9 Recursive Approaches
The following methods recursively apply a Monte Carlo
technique to grow the search tree. These have typically
had success with single-player puzzles and similar optimisation
tasks.
4.9.1 Reflexive Monte Carlo Search
Reflexive Monte Carlo search [39] works by conducting
several recursive layers of Monte Carlo simulations, each
layer informed by the one below. At level 0, the simulations
simply use random moves (so a level 0 reflexive
Monte Carlo search is equivalent to a 1-ply search with
Monte Carlo evaluation). At level n > 0, the simulation
uses level n − 1 searches to select each move.
4.9.2 Nested Monte Carlo Search
A related algorithm to reflexive Monte Carlo search is
nested Monte Carlo search (NMCS) [42]. The key difference
is that nested Monte Carlo search memorises the best
sequence of moves found at each level of the search.
Memorising the best sequence so far and using this
knowledge to inform future iterations can improve the
performance of NMCS in many domains [45]. Cazenaze
et al. describe the application of NMCS to the bus
regulation problem (7.8.3) and find that NMCS with
memorisation clearly outperforms plain NMCS, which
in turn outperforms flat Monte Carlo and rule-based
approaches [45].
Cazenave and Jouandeau [49] describe parallelised
implementations of NMCS. Cazenave [43] also demonstrates
the successful application of NMCS methods
for the generation of expression trees to solve certain
mathematical problems (7.8.2). Rimmel et al. [168] apply
a version of nested Monte Carlo search to the Travelling
Salesman Problem (TSP) with time windows (7.8.1).