Neural Models of Bayesian Belief Propagation Rajesh ... - Washington

242 11 Neural Models of Bayesian Belief Propagation Rajesh P. N. Rao
time step, one can relax the equality in equation (11.21) and make log P (I ′ |θt i ) ∝
F(θi)I(t) for some linear filter F(θi) = ɛwi. This avoids the problem of calculating
the normalization factor for P (I ′ |θt i ), which can be especially hard when I′
takes on continuous values such as in an image. A more challenging problem
is to pick recurrent weights Uij such that equation (11.22) holds true. For equation
(11.22) to hold true, we need to approximate a log-sum with a sum-of-logs.
One approach is to generate a set of random probabilities xj(t) for t = 1, . . . , T
and find a set of weights Uij that satisfy:
j
Uij log xj(t) ≈ log P (θ t i|θ t−1
j )xj(t)
j
(11.24)
for all i and t. This can be done by minimizing the squared error in equation
(11.24) with respect to the recurrent weights Uij. This empirical approach,
followed in [30], is used in some of the experiments below. An alternative approach
is to exploit the nonlinear properties of dendrites as suggested in the
following section.
11.3.2 Exact Inference in Nonlinear Networks
A firing rate model that takes into account some of the effects of nonlinear filtering
in dendrites can be obtained by generalizing equation (11.18) as follows:
vi(t + 1) = f wiI(t) + g
Uijvj(t) , (11.25)
j
where f and g model nonlinear dendritic filtering functions for feedforward
and recurrent inputs. By comparing this equation with the belief propagation
equation in the log domain (equation (11.19)), it can be seen that the first equation
can implement the second if:
vi(t + 1) = log m t i (11.26)
f wiI(t) = log P (I ′ |θ t i) (11.27)
g
Uijvj(t) = log
P (θ t i|θ t−1
j )m t−1,t
j
(11.28)
j
j
In this model (figure 11.2B), N neurons represent log mt i (i = 1, . . . , N) in their
firing rates. The dendritic filtering functions f and g approximate the logarithm
function, the feedforward weights wi act as a linear filter on the input
to yield the likelihood P (I ′ |θt i ) and the recurrent synaptic weights Uij directly
encode the transition probabilities P (θt i |θt−1 j ). The normalization step is com-
puted as in equation (11.23) using a separate group of neurons that represent
log posterior probabilities log m t,t+1
i and that convey these probabilities for use
.
in equation (11.28) by the neurons computing log m t+1
i