Chapter 2. Prehension

Appendix C - Computational Neural Modelling 397
O=WI (16)
Now, if there are p input/output pairs, we can define two n x p
matrices, 1. and 0, where the columns of 1. are the p input patterns and
the columns of 0 are the corresponding p output patterns. Then we
have
for correct performance. But this equation also defines W and
answers our first question. By inverting it we obtain:
which gives an explicit formula for computing W (although it is
necessary to perform the time consuming task of inverting I.) It also
provides an upper limit on p (the memory’s capacity), answering the
second question: I is only guaranteed to be invertible if pln. For the
remainder of the discussion we will assume p=n, that is the matrices
are square. The input patterns must then be linearly independent,
since I must be invertible (its determinant must be non-zero).
Computing W can be time consuming, as mentioned above, but a
special case occurs when all the input patterns are orthonormal. In this
case, L-l =LT, and the computation of W simply requires a single
matrix multiplication. The orthonormal condition is often assumed
true when the patterns are binary, long (large n), and sparse (mostly
zero), because they tend to be orthogonal (their dot products are likely
to be zero). If they all have about the same number of one’s, then they
can all be scaled down by that number to be roughly normal (of unit
magnitude). The resulting performance of the memory is close to
correct. For non-orthonormal situations, there is the option of
applying a gradient descent technique (e.g. the delta rule) to iteratively
learn the weight matrix.
C.4 Processing Example
As was shown in the previous section, a heteroassociative memory
allows two patterns to be associated with one another, such that one
pattern (the input pattern) triggers the other pattern to appear at the
output units. For this example, a network with three inputs and three
outputs (all linear) will be used, as seen in Figure C.9. Assume your