MACHINE LEARNING TECHNIQUES - LASA

173
general, acting to maximize immediate reward can reduce access to future rewards so that the
return may actually be reduced. As γ approaches 1, the objective takes future rewards into
account more strongly: the agent becomes more farsighted. We can interpret γ in several ways.
It can be seen as an interest rate, a probability of living another step, or as a mathematical trick to
bound the infinite sum.
Another optimality criterion is the average-reward model, in which the agent is supposed to take
actions that optimize its long-run average reward:
⎛⎛1
lim E
h→∞
⎜⎜
⎝⎝h
h
⎞⎞
r ⎟⎟
⎠⎠
∑ (7.17)
t
t=
0
Such a policy is referred to as a gain optimal policy; it can be seen as the limiting case of the
infinite-horizon discounted model as the discount factor approaches. One problem with this
criterion is that there is no way to distinguish between two policies, one of which gains a large
amount of reward in the initial phases and the other of which does not. Reward gained on any
initial prefix of the agent's life is overshadowed by the long-run average performance. It is
possible to generalize this model so that it takes into account both the long run average and the
amount of initial reward than can be gained. In the generalized, bias optimal model, a policy is
preferred if it maximizes the long-run average and ties are broken by the initial extra reward.
7.3.3 Markov World
A reinforcement-learning task is described as a Markov decision process, or MDP. If the state
and action spaces are finite, then it is called a finite Markov decision process (finite MDP).
A particular finite MDP is defined by its state and action sets and by the one-step dynamics of the
environment. Given any state and action, s and a, the probability of each possible next state, s’, is
{ }
P = P s = s'| s = s,
a = a
(7.18)
a
ss' t+
1 t t
These quantities are called transition probabilities. Similarly, given any current state and action,
s and a , together with any next state, s’, the expected value of the next reward is
These quantities,
P and
a
ss'
{ | , , '}
R = E r s = s a = a s = s (7.19)
a
ss' t+ 1 t t t+
1
R
a
ss'
, completely specify the most important aspects of the dynamics of
a finite MDP (only information about the distribution of rewards around the expected value is lost).
Almost all reinforcement learning algorithms are based on estimating value functions--functions of
states (or of state-action pairs) that estimate how good it is for the agent to be in a given state (or
how good it is to perform a given action in a given state). The notion of "how good" here is
© A.G.Billard 2004 – Last Update March 2011