Pattern recognition in the neocognitron is improved by ... - Springer
Pattern recognition in the neocognitron is improved by ... - Springer
Pattern recognition in the neocognitron is improved by ... - Springer
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
Biol. Cybern. 70, 47-53 (1993) Bkalog ,eOybemetics9 Spr<strong>in</strong>ger-Verlag 1993<strong>Pattern</strong> <strong>recognition</strong> <strong>in</strong> <strong>the</strong> <strong>neocognitron</strong> <strong>is</strong> <strong>improved</strong> <strong>by</strong> neuronal adaptationA. van Ooyen 1, B. Nienhu<strong>is</strong> ~Ne<strong>the</strong>rlands Institute for Bra<strong>in</strong> Research, Meibergdreef 33, 1105 AZ Amsterdam, The Ne<strong>the</strong>rlandsInstitute for Theoretical Physics, University of Amsterdam, Valckenierstraat 65, 1018 XE Amsterdam, The Ne<strong>the</strong>rlandsReceived: 2 October 1992/Accepted <strong>in</strong> rev<strong>is</strong>ed form: 7 June 1993Abstract. We demonstrate that equipp<strong>in</strong>g <strong>the</strong> neuronsof Fukushima's <strong>neocognitron</strong> with <strong>the</strong> phenomenonthat a neuron decreases its activity when repeatedlystimulated (adaptation) markedly improves <strong>the</strong> patternd<strong>is</strong>crim<strong>in</strong>atory power of <strong>the</strong> network. By means ofadaptation, circuits for extract<strong>in</strong>g d<strong>is</strong>crim<strong>in</strong>at<strong>in</strong>g featuresdevelop preferentially. In <strong>the</strong> orig<strong>in</strong>al <strong>neocognitron</strong>, <strong>in</strong>contrast, features shared <strong>by</strong> different patterns are preferentiallylearned, as connections required for extract<strong>in</strong>g<strong>the</strong>m are more frequently re<strong>in</strong>forced.1 IntroductionIn most neural network models, <strong>the</strong> <strong>in</strong>dividual units areover-simplified compared with biotic neurons. Usually,only <strong>the</strong> very basic <strong>in</strong>put-output properties are considered.In <strong>the</strong> simplest case, each unit can be <strong>in</strong> twostates only: ei<strong>the</strong>r it <strong>is</strong> at rest or it fires at its maximumfrequency.The o<strong>the</strong>r extreme <strong>is</strong> formed <strong>by</strong> network models <strong>in</strong>which one attempts to model <strong>the</strong> constituent neurons asreal<strong>is</strong>tically as possible, even down to <strong>the</strong> level of ionicconductances <strong>in</strong> different neuron com-partments (e.g.,Traub 1982). Because of <strong>the</strong> complexity of such models, it<strong>is</strong> usually not attempted to p<strong>in</strong>po<strong>in</strong>t <strong>the</strong> role <strong>the</strong> variousneuronal properties play <strong>in</strong> <strong>the</strong> behavior of <strong>the</strong> network.In general, such models are too complicated to be ofmuch help <strong>in</strong> answer<strong>in</strong>g <strong>the</strong> question of how variousneuronal properties might contribute to <strong>the</strong> <strong>in</strong>formationprocess<strong>in</strong>gpower of neural networks. In order to establ<strong>is</strong>hth<strong>is</strong>, one should work with series of models of vary<strong>in</strong>gcomplexity. The importance of particular properties can<strong>the</strong>n be establ<strong>is</strong>hed <strong>by</strong> add<strong>in</strong>g (or remov<strong>in</strong>g) <strong>the</strong>m from<strong>the</strong> model (see De Boer 1989).In th<strong>is</strong> paper, we focus on one such neuronal property,namely adaptation, and study what role it couldplay <strong>in</strong> pattern <strong>recognition</strong>. The term adaptation <strong>is</strong> usedhere to describe <strong>the</strong> phenomenon that a neuron becomesCorrespondence to: A. van Ooyengradually less responsive when repeatedly stimulated,reflected <strong>by</strong> a decrease <strong>in</strong> its fir<strong>in</strong>g frequency. Forexample, neurons <strong>in</strong> <strong>the</strong> <strong>in</strong>ferior temporal cortex <strong>in</strong>monkeys give strong responses to v<strong>is</strong>ual stimuli that arenew or that have not recently been seen (Rolls et al. 1989;Miller et al. 1991). Possible mechan<strong>is</strong>ms for adaptation<strong>in</strong>clude an <strong>in</strong>creas<strong>in</strong>g potassium conductance as a resultof calcium <strong>in</strong>flux dur<strong>in</strong>g susta<strong>in</strong>ed depolarization (Meech1978; Connors et al. 1982). In Desimone (1992) adaptationor adaptive filter<strong>in</strong>g <strong>is</strong> mentioned as one of <strong>the</strong>neuronal mechan<strong>is</strong>ms that may play a role <strong>in</strong> <strong>the</strong> formationof memory traces. In order to study how adaptationcould contribute to pattern <strong>recognition</strong>, we have<strong>in</strong>corporated it <strong>in</strong>to <strong>the</strong> <strong>neocognitron</strong>, which <strong>is</strong> a multilayeredfeedforward neural network model for v<strong>is</strong>ualpattern <strong>recognition</strong> (Fukushima 1980; Fukushima andMiyake 1982; Miyake and Fukushima 1984; Fukushima1988; Menon and He<strong>in</strong>emann 1988; Fukushima 1989).The <strong>neocognitron</strong> has <strong>the</strong> capacity to carry out translationand size <strong>in</strong>variant pattern <strong>recognition</strong> (but see Barnardand Casasent 1990), and <strong>is</strong> tra<strong>in</strong>ed <strong>by</strong> nonsuperv<strong>is</strong>edlearn<strong>in</strong>g. The learn<strong>in</strong>g mechan<strong>is</strong>m employed <strong>is</strong>a type of competitive learn<strong>in</strong>g I-networks based on competitivelearn<strong>in</strong>g can be found <strong>in</strong>, e.g., Kohonen (1984),Carpenter and Grossberg (1987), and Reilly et al. (1982)].After <strong>the</strong> learn<strong>in</strong>g process <strong>is</strong> completed, <strong>the</strong> model hasa hierarchical structure <strong>in</strong> which simple features are comb<strong>in</strong>edstep <strong>by</strong> step <strong>in</strong>to more complicated features. Thetun<strong>in</strong>g of <strong>the</strong> network can be complicated because of <strong>the</strong>large number of parameters. Recently it has been shownthat <strong>the</strong> model can be greatly simplified while preserv<strong>in</strong>gits <strong>in</strong>terest<strong>in</strong>g properties of self-organization and toleranceof image deformations (Trot<strong>in</strong> et al. 1991). The<strong>neocognitron</strong> can be used for various practical applications(e.g., Fukushima 1988; Fukushima and Imagawa1993).The organization of th<strong>is</strong> article <strong>is</strong> as follows. First, <strong>the</strong>orig<strong>in</strong>al model will be described. Next, <strong>the</strong> modifiedversion will be given <strong>in</strong> which adaptation <strong>is</strong> <strong>in</strong>corporated.The effects of adaptation on <strong>the</strong> developmentof feature-detect<strong>in</strong>g circuits will <strong>the</strong>n be described.Subsequently, <strong>the</strong> performance of <strong>the</strong> network, with andwithout adaptation, will be compared <strong>in</strong> simulationexperiments.
482 The <strong>neocognitron</strong>In th<strong>is</strong> section we will briefly describe <strong>the</strong> structure of<strong>the</strong> network (after learn<strong>in</strong>g <strong>is</strong> completed) and <strong>the</strong> selforganization(learn<strong>in</strong>g) of <strong>the</strong> network. For a more detaileddescription <strong>the</strong> reader <strong>is</strong> referred to Fukushimaand Miyake (1982) and <strong>the</strong> Appendix.2.1 Structure of <strong>the</strong> networkAfter completion of <strong>the</strong> learn<strong>in</strong>g process, <strong>the</strong> <strong>neocognitron</strong>has a structure <strong>in</strong> which cells <strong>in</strong> <strong>the</strong> lowest levelextract local features of <strong>the</strong> <strong>in</strong>put pattern, while cells <strong>in</strong>each succeed<strong>in</strong>g level respond to specific comb<strong>in</strong>ations of<strong>the</strong> features detected <strong>in</strong> <strong>the</strong> preced<strong>in</strong>g level. In <strong>the</strong> highestlevel, each cell will respond to only one <strong>in</strong>put pattern, i.e.,<strong>the</strong> pattern <strong>is</strong> recognized. Th<strong>is</strong> structure <strong>is</strong> similar to <strong>the</strong>hierarchical model of <strong>the</strong> v<strong>is</strong>ual system proposed <strong>by</strong>Hubel and Wiesel (1962, 1965).The <strong>in</strong>put layer <strong>is</strong> composed of a two-dimensionalarray of receptor cells. Each of <strong>the</strong> succeed<strong>in</strong>g levelscons<strong>is</strong>ts of a layer of excitatory S-cells (S-layer) followed<strong>by</strong> a layer of excitatory C-cells (C-layer; see Fig. 1). EachC-layer also conta<strong>in</strong>s <strong>in</strong>hibitory V-cells (not shown <strong>in</strong><strong>the</strong> figure). An S-cell receives excitatory connectionsfrom a certa<strong>in</strong> group of C-ceUs <strong>in</strong> <strong>the</strong> preced<strong>in</strong>g level. AnS-cell also receives an <strong>in</strong>hibitory connection from a V-cell, which <strong>in</strong> turn receives fixed excitatory connectionsfrom <strong>the</strong> same group of C-cells as does <strong>the</strong> S-cell towhich it projects. After <strong>the</strong> learn<strong>in</strong>g stage <strong>is</strong> f<strong>in</strong><strong>is</strong>hed,S-cells extract features from <strong>the</strong> <strong>in</strong>put patterns. With<strong>in</strong><strong>the</strong>ir respective layers, S-cells and C-cells are divided<strong>in</strong>to cell-planes. All <strong>the</strong> cells <strong>in</strong> such a cell-plane extract<strong>the</strong> same feature but from different positions of <strong>the</strong><strong>in</strong>put layer, while different cell-planes extract differentfeatures. Each C-cell receives signals from a group ofpreced<strong>in</strong>g S-cells, all of which extract identical featuresbut from slightly different positions. A C-cell will beactivated if at least one of <strong>the</strong>se S-cells <strong>is</strong> active, and <strong>is</strong><strong>the</strong>refore less sensitive to positional shifts of <strong>the</strong> <strong>in</strong>putpattern than <strong>is</strong> an S-cell. The size of cell-planes<strong>in</strong> both S- and C-layers decreases with <strong>the</strong> order of <strong>the</strong>level, and <strong>in</strong> <strong>the</strong> highest level each cell-plane <strong>in</strong> <strong>the</strong>C-layer has only one C-cell which responds, if <strong>the</strong> learn<strong>in</strong>ghas been successful, to only one particular <strong>in</strong>putpattern.2.2 Learn<strong>in</strong>gOnly <strong>the</strong> excitatory and <strong>in</strong>hibitory connections to S-cells are variable. To modify <strong>the</strong>m, maximum-outputcells (MOCs) are chosen <strong>in</strong> <strong>the</strong> follow<strong>in</strong>g way. They areselected from each S-layer every time a stimulus pattern<strong>is</strong> presented. In an S-layer, from a group of cells thatextract features with<strong>in</strong> a small area on <strong>the</strong> <strong>in</strong>put layer(such a group <strong>is</strong> called a hypercolumn; see Fig. 2), <strong>the</strong> cellwith <strong>the</strong> largest positive response <strong>is</strong> chosen as a candidatefor <strong>the</strong> MOC. Th<strong>is</strong> <strong>is</strong> repeated for different such hypercolumns,each of which receives signals from differentareas of <strong>the</strong> <strong>in</strong>put layer. If more than one candidate hasbeen chosen <strong>in</strong> <strong>the</strong> same cell-plane, <strong>the</strong> one with <strong>the</strong>largest response <strong>is</strong> selected. All <strong>the</strong> MOCs from differenthypercolumns have <strong>the</strong>ir <strong>in</strong>put connections, both excitatoryand <strong>in</strong>hibitory, re<strong>in</strong>forced. The amount of re<strong>in</strong>forcement<strong>is</strong> proportional to <strong>the</strong> signal strength througha connection, so that <strong>the</strong> excitatory connections willgrow to match <strong>the</strong> response <strong>in</strong> <strong>the</strong> preced<strong>in</strong>g layer (i.e.,<strong>the</strong> S-cell will detect <strong>the</strong> feature that has been presented).All <strong>the</strong> o<strong>the</strong>r S-cells <strong>in</strong> <strong>the</strong> cell-plane from which <strong>the</strong>MOC has been selected have <strong>the</strong>ir connections re<strong>in</strong>forced<strong>in</strong> <strong>the</strong> same way as <strong>the</strong> MOC, so that <strong>the</strong>y all cometo respond to <strong>the</strong> same feature but are extracted fromdifferent positions of <strong>the</strong> <strong>in</strong>put layer. When a novel feature<strong>is</strong> presented to <strong>the</strong> same S-cell, <strong>the</strong> <strong>in</strong>hibitory <strong>in</strong>putfrom <strong>the</strong> V-cell <strong>is</strong> stronger than <strong>the</strong> excitatory <strong>in</strong>put, as<strong>the</strong> excitatory connections do not optimally match th<strong>is</strong>feature. Usually, an S-cell <strong>in</strong> ano<strong>the</strong>r cell-plane <strong>is</strong> nowan MOC and has its connections re<strong>in</strong>forced. Thus, S-cells <strong>in</strong> different cell-planes come to respond to differentfeatures. Th<strong>is</strong> process of select<strong>in</strong>g MOCs and re<strong>in</strong>forc<strong>in</strong>gconnections <strong>is</strong> repeated for each level of <strong>the</strong> network andfor each time an <strong>in</strong>put pattern <strong>is</strong> presented.cell-plmw~ !- hL\ ,_c ,.i )\ :IJ ,S-layer ~......~ttL- ~Fig. 1. Schematic diagram illustrat<strong>in</strong>g <strong>the</strong> structure of <strong>the</strong> <strong>neocognitron</strong>.Modified from Fukushima (1980)Fig. 2. Hypercolumns with<strong>in</strong> an S-layer. Modified from Fukushima(1980)
- T(k).493 Incorporat<strong>in</strong>g adaptationIn <strong>the</strong> orig<strong>in</strong>al <strong>neocognitron</strong>, circuits detect<strong>in</strong>g featurescommon among a set of <strong>in</strong>put patterns develop <strong>in</strong> preferenceto those detect<strong>in</strong>g rare or unique features. S-cellsrespond<strong>in</strong>g to common features will have <strong>the</strong>ir <strong>in</strong>putconnections more often re<strong>in</strong>forced than o<strong>the</strong>r S-cells, andwill <strong>the</strong>refore get higher responses. Cells <strong>in</strong> succeed<strong>in</strong>glevels will preferentially have those <strong>in</strong>put connectionsre<strong>in</strong>forced that come from cells which are frequentlyactivated and which have high levels of activity. Hence,strong connections will develop to such cells, which alsodetect, however, <strong>the</strong> least d<strong>is</strong>crim<strong>in</strong>at<strong>in</strong>g features for pattern<strong>recognition</strong>. Consequently, <strong>the</strong> <strong>in</strong>formation passedon to higher levels will not always be specific enough for<strong>the</strong> C-cells <strong>in</strong> <strong>the</strong> highest level to respond selectively toonly one particular <strong>in</strong>put pattern. In what follows, wewill show that neuronal adaptation, i.e., a cell becom<strong>in</strong>gless responsive when frequently stimulated, improvespattern selectivity.Adaptation <strong>is</strong> implemented <strong>by</strong> modify<strong>in</strong>g <strong>the</strong> responsecharacter<strong>is</strong>tic of <strong>the</strong> C-cells (Eq. 8 <strong>in</strong> Appendix),which <strong>by</strong> add<strong>in</strong>g only one term to <strong>the</strong> denom<strong>in</strong>atorbecomes:I 1 + e(k, n) 1Ucz(k,n)=~b l+h(k,n)+ T(k)- I (1)where e(k, n) and h(k, n) are def<strong>in</strong>ed as <strong>the</strong> excitatory and<strong>in</strong>hibitory effect, respectively (for <strong>the</strong> explanation of <strong>the</strong>o<strong>the</strong>r symbols, see Appendix):e(k, n) = ~ d,(v)- usdk, n + v) (2)vEDf1 Kh(k, n) = ~ k~ ' ~ d~(v)'usl(k, n + v) (3)= vCDtand T(k) <strong>is</strong> a variable controll<strong>in</strong>g <strong>the</strong> degree of adaptation.At a given time, all <strong>the</strong> C-cells of <strong>the</strong> kthcell-plane of <strong>the</strong> /th level have <strong>the</strong> same degreeof adaptation. The greater <strong>the</strong> value of T, <strong>the</strong> moreadapted a C-cell <strong>is</strong> and <strong>the</strong> weaker its response to <strong>the</strong>feature which it detects via <strong>the</strong> S-cells <strong>in</strong> <strong>the</strong> preced<strong>in</strong>glayer. The first term <strong>in</strong> <strong>the</strong> argument of ~b (Eq. 1) can bethought of as <strong>the</strong> membrane potential of <strong>the</strong> neuron,which <strong>is</strong> ra<strong>is</strong>ed <strong>by</strong> excitatory <strong>in</strong>puts and shunted <strong>by</strong>lateral <strong>in</strong>hibition and adaptation. The term Uc~ represents<strong>the</strong> fir<strong>in</strong>g frequency; it <strong>is</strong> proportional to <strong>the</strong> differencebetween membrane potential and rest<strong>in</strong>g potential,<strong>the</strong> latter be<strong>in</strong>g represented <strong>by</strong> <strong>the</strong> second term <strong>in</strong> <strong>the</strong>argument of ~9.Before <strong>the</strong> update rule for T will be given, we rewrite(1) as:Uc~(k'n)=O[ e(k'n)-h(k'n)-~T(k)~+-h(k,~+ T(k) J (4)Each time an <strong>in</strong>put pattern <strong>is</strong> presented to <strong>the</strong> network,T (k) <strong>is</strong> updated <strong>by</strong> <strong>the</strong> follow<strong>in</strong>g amount:AT(k) = g~" {Pt" q) [e.,(k) - hm(k)] - T(k)} (5)where T (k) <strong>is</strong> <strong>the</strong> degree of adaptation of all <strong>the</strong> C-cells <strong>in</strong><strong>the</strong> kth cell-plane of <strong>the</strong>/th level; g~ controls <strong>the</strong> speed ofadaptation <strong>in</strong> <strong>the</strong>/th level; p, determ<strong>in</strong>es <strong>the</strong> maximumvalue of T(k) under susta<strong>in</strong>ed stimulation; e,,(k) <strong>is</strong> <strong>the</strong>maximum value of e(k, n) found among <strong>the</strong> C-cells of <strong>the</strong>kth cell-plane; h,,(k) <strong>is</strong> <strong>the</strong> maximum value of h(k, n);q0[x] = x if x >~ 0 and (p[x] = 0 if x < 0.Equations 4 and 5 have <strong>the</strong> follow<strong>in</strong>g implications.Consider <strong>the</strong> first level of <strong>the</strong> network (i,e., l -- 1), whichextracts simple features from <strong>the</strong> <strong>in</strong>put pattern. Supposethat <strong>the</strong> feature detected <strong>by</strong> cell-plane k <strong>is</strong> present <strong>in</strong> all<strong>in</strong>put patterns, and that at least one of <strong>the</strong> C-cells <strong>in</strong> <strong>the</strong>kth cell-plane <strong>is</strong> activated each time an <strong>in</strong>put pattern<strong>is</strong> presented, i.e., ~p[e,,(k)-h,,(k)] <strong>is</strong> always greaterthan zero (T(k)=0 <strong>in</strong>itially). Fur<strong>the</strong>r assume that~p[em(k)-hm(k)] <strong>is</strong> <strong>the</strong> same for each <strong>in</strong>put pattern.Then, <strong>the</strong> value of T(k) will <strong>in</strong>crease until it equalsPt'r [era(k) - h,,(k)], and <strong>the</strong> response of all C-cells <strong>in</strong> <strong>the</strong>kth cell-plane becomes strongly depressed. If p~ = 1, <strong>the</strong>response will become zero.Now consider a cell-plane respond<strong>in</strong>g to a featurethat occurs only <strong>in</strong> a subset of <strong>the</strong> <strong>in</strong>put patterns. Ifa pattern conta<strong>in</strong><strong>in</strong>g th<strong>is</strong> feature <strong>is</strong> presented, and <strong>the</strong>cell-plane <strong>is</strong> activated <strong>by</strong> it, adaptation will be build up<strong>by</strong> an amount proportional to pl'~p[e,,(k)--hm(k)]If <strong>the</strong> next <strong>in</strong>put patterns do not conta<strong>in</strong> th<strong>is</strong>feature (i.e., (p[e,,(k)- h,,(k)] = 0), T(k) decreases <strong>by</strong>g~" T (k). If <strong>the</strong> set of <strong>in</strong>put patterns <strong>is</strong> presented repeatedly,and g~ <strong>is</strong> small enough, T(k) will converge to <strong>the</strong>average of p~'q)[e,,(k) - h,,(k)] elicited <strong>by</strong> different <strong>in</strong>putpatterns. Thus, <strong>the</strong> equilibrium value of T(k) will bedeterm<strong>in</strong>ed <strong>by</strong> feature frequency: <strong>the</strong> greater <strong>the</strong> frequency,<strong>the</strong> higher <strong>the</strong> equilibrium value of T (k) and <strong>the</strong>weaker <strong>the</strong> response.Now consider <strong>the</strong> case <strong>in</strong> which <strong>the</strong> feature detected<strong>by</strong> cell-plane k <strong>is</strong> present <strong>in</strong> all <strong>in</strong>put patterns but doesnot produce exactly <strong>the</strong> same response each time, e.g.,due to lateral <strong>in</strong>hibition of o<strong>the</strong>r cell-planes. That <strong>is</strong>,~p [em(k) - hr,(k)] <strong>is</strong> always greater than zero but does notalways have <strong>the</strong> same value each time an <strong>in</strong>put pattern <strong>is</strong>presented. After T (k) has reached its equilibrium value,cell-plane k <strong>is</strong> only active if <strong>the</strong> response <strong>is</strong> so strong thatit exceeds T(k).Adaptation has <strong>the</strong> follow<strong>in</strong>g effect on <strong>the</strong> developmentof connections lead<strong>in</strong>g to succeed<strong>in</strong>g levels. Because<strong>the</strong> response of C-cells activated <strong>by</strong> frequently occurr<strong>in</strong>gfeatures <strong>is</strong> weak due to adaptation, connections lead<strong>in</strong>gfrom <strong>the</strong>se cells to succeed<strong>in</strong>g S-cells will not be preferentiallyre<strong>in</strong>forced. Connections lead<strong>in</strong>g from C-cells thatrespond to rare or unique features, on <strong>the</strong> o<strong>the</strong>r hand, arepreferentially re<strong>in</strong>forced because <strong>the</strong> output of <strong>the</strong>se cells<strong>is</strong> relatively high due to a low degree of adaptation. Inth<strong>is</strong> way, circuits for detect<strong>in</strong>g d<strong>is</strong>crim<strong>in</strong>at<strong>in</strong>g featureswill develop preferentially.Cells <strong>in</strong> higher levels than <strong>the</strong> first comb<strong>in</strong>e simplefeatures extracted <strong>by</strong> cells <strong>in</strong> lower levels. Therefore, cells<strong>in</strong> higher levels will become differentially adapted to
50comb<strong>in</strong>ations of simple features. The more frequentlya particular comb<strong>in</strong>ation occurs, <strong>the</strong> higher <strong>the</strong> degree ofadaptation for cells detect<strong>in</strong>g th<strong>is</strong> comb<strong>in</strong>ation.a4 Simulation resultsIn th<strong>is</strong> section, some experiments with <strong>the</strong> modified<strong>neocognitron</strong> will be presented, <strong>the</strong> results of which willbe compared with those obta<strong>in</strong>ed with <strong>the</strong> orig<strong>in</strong>al one.~N~DDDDmD4.1 Parameter sett<strong>in</strong>gsA reasonable parameter sett<strong>in</strong>g for a range of experiments<strong>in</strong>volv<strong>in</strong>g <strong>recognition</strong> of numerals was chosen. Tofacilitate compar<strong>is</strong>on, <strong>the</strong> same sett<strong>in</strong>g was used <strong>in</strong> <strong>the</strong>orig<strong>in</strong>al and modified <strong>neocognitron</strong>. The network usedhas n<strong>in</strong>e layers: an <strong>in</strong>put layer followed <strong>by</strong> four levels,each cons<strong>is</strong>t<strong>in</strong>g of an S- and a C-layer. The first level has10 cell-planes, whereas <strong>the</strong> three o<strong>the</strong>r levels each have 15cell-planes. The number of cells <strong>in</strong> <strong>the</strong> <strong>in</strong>put layer <strong>is</strong>16 • 16. The number of cells <strong>in</strong> a given cell-plane <strong>is</strong>:16 x 16 <strong>in</strong> Us1 (S-layer of level 1); 8 x 8 <strong>in</strong> Ucl (C-layer oflevel 1) and Us2; 4 x 4 <strong>in</strong> Uc2 and Us3; 2 x 2 <strong>in</strong> Uc3 andUs4; and 1 <strong>in</strong> Uc4. In every layer, <strong>the</strong> connect<strong>in</strong>g area S~<strong>is</strong> 5 x 5 and D~ <strong>is</strong> 6 • 6. Parts of <strong>the</strong> connection area fall<strong>in</strong>goutside a layer receive zero <strong>in</strong>put. The value of r~, whichcontrols <strong>the</strong> <strong>in</strong>tensity of <strong>the</strong> <strong>in</strong>hibitory <strong>in</strong>put to an S-cell,decreases with <strong>the</strong> order of <strong>the</strong> level: r~ = 8.0, r2 = 5.8,r 3 = 5.0, and r4 = 4.5. The degree of saturation (cq) <strong>is</strong>:~ = 0.1, e2 = 0.004, a3 = 0.02, and :t4 = 0.3. The speedof re<strong>in</strong>forcement of <strong>the</strong> variable connections, q~, <strong>in</strong>creaseswith <strong>the</strong> order of <strong>the</strong> level: q~ = 1.3, q2 = 2.3, q3 = 3.3,and q4 = 4.3. The value of m, which determ<strong>in</strong>es <strong>the</strong> durationof <strong>the</strong> connections a and b (see Appendix) decreaseswith <strong>the</strong> order of <strong>the</strong> level: m~ = 35, m2 = 30, m3 = 28,and m, = 25. The values of <strong>the</strong> fixed connections c andd are determ<strong>in</strong>ed as <strong>in</strong> Fukushima and Miyake (1982).4.2 Recognition of four numeralsIn order to study closely <strong>the</strong> effects of adaptation andto demonstrate what factors can hamper pattern <strong>recognition</strong><strong>in</strong> <strong>the</strong> orig<strong>in</strong>al <strong>neocognitron</strong>, we focus on anexperiment <strong>in</strong> which only four numerals with commonelementary features were presented. The <strong>in</strong>put patterns 8,5, 6, and 9 (see Figs. 3, 4) were presented <strong>in</strong> a cyclicalmanner, i.e., 85698569 .... The positions of <strong>the</strong>se patternswere not shifted dur<strong>in</strong>g presentation.4.2.1 Orig<strong>in</strong>al <strong>neocognitron</strong>. In th<strong>is</strong> small experiment,a network cons<strong>is</strong>t<strong>in</strong>g of an <strong>in</strong>put layer followed <strong>by</strong> onlythree levels would have been sufficient, as <strong>the</strong>re turnedout to be a one-to-one correspondence between <strong>the</strong> activationpattern <strong>in</strong> <strong>the</strong> third and <strong>the</strong> fourth level. With<strong>the</strong> parameter sett<strong>in</strong>g used, <strong>the</strong> network did not acquire<strong>the</strong> ability to d<strong>is</strong>t<strong>in</strong>gu<strong>is</strong>h 5 and 6: <strong>in</strong> <strong>the</strong> highest layer, oneand <strong>the</strong> same C-cell <strong>is</strong> activated <strong>by</strong> both numerals. Although<strong>the</strong> features with which <strong>the</strong>y can be d<strong>is</strong>t<strong>in</strong>gu<strong>is</strong>hedDDmDb ~ ~ ~o1 ,5 10rO~SD~DDDSD@DSDNDSDD~~D~DDNDDD~3DD~NDDD~r,[i3~D@DIB]DDU]DE3NID~DDDDmd 1 5I[][]DD[]DDDSD[]D[]DD4DDD~DFig. 3a-d. Neocognitron without adaptation, a Response of <strong>the</strong>cells <strong>in</strong> Ucl, Uc2, and Uc3 to each of <strong>the</strong> four <strong>in</strong>put patterns.Note that <strong>the</strong> number<strong>in</strong>g of <strong>the</strong> cell planes <strong>in</strong> Ucl corresponds to <strong>the</strong>feature number<strong>in</strong>g <strong>in</strong> b; b features extracted <strong>by</strong> <strong>the</strong> S-cells <strong>in</strong> <strong>the</strong>first level; e connect<strong>in</strong>g areas between S-layer 2 (vertical) and C-layer1 (horizontal); d connect<strong>in</strong>g areas between S-layer 3 (vertical) andC-layer 2 (horizontal). In e and d, only high connection strengths areshownare extracted <strong>by</strong> cells <strong>in</strong> <strong>the</strong> first level, it <strong>is</strong> ma<strong>in</strong>ly <strong>the</strong>features shared <strong>by</strong> both patterns that are extracted <strong>by</strong>cells <strong>in</strong> <strong>the</strong> second level, as can be seen <strong>in</strong> Fig. 3a. Asa result, 5 and 6 cannot be d<strong>is</strong>t<strong>in</strong>gu<strong>is</strong>hed. (Because <strong>in</strong> th<strong>is</strong>case only a few patterns needed to be recognized, it wouldbe possible to recognize all patterns <strong>by</strong> choos<strong>in</strong>g a differentparameter sett<strong>in</strong>g.)4.2.2 Neocognitron with adaptation. The <strong>in</strong>itial degree ofadaptation <strong>in</strong> each cell-plane was set at zero. There wasadaptation only <strong>in</strong> <strong>the</strong> first level, i.e., 92 = g3 = 94 = O.The speed of adaptation <strong>in</strong> <strong>the</strong> first level, gl, was set at0.05; and pt, which determ<strong>in</strong>es <strong>the</strong> maximum equilibriumvalue of T(k), was set at 1.0. With th<strong>is</strong> value, T(k)stabilized around an equilibrium value after about 100presentations of <strong>the</strong> whole set of <strong>in</strong>put patterns. BecauseT of each cell-plane converges to <strong>the</strong> average of <strong>the</strong>responses elicited <strong>by</strong> different numerals, cell-planeswill be activated only if <strong>the</strong> response exceeds <strong>the</strong>adaptation level. Therefore, only a reduced set of featureswill be active for each <strong>in</strong>put pattern: <strong>the</strong> d<strong>is</strong>crim<strong>in</strong>at<strong>in</strong>gfeatures (i.e., <strong>in</strong>frequent features) are enhanced (see Fig.4). As a result, 5 and 6 are now recognized as be<strong>in</strong>gdifferent.The feature with which 5 can be d<strong>is</strong>t<strong>in</strong>gu<strong>is</strong>hed fromall o<strong>the</strong>r numerals (feature 3; see Fig. 4b), <strong>is</strong> now used <strong>in</strong><strong>the</strong> circuit for recogniz<strong>in</strong>g 5.
51a@~@@@b 1 5 10c , 5 lO dDD~5~DD~@~N@D89N~@g89GNDG[]~@@NN,oN9C~C~ U] m1 ,5 10D~E3[][]D[][]D~5E3~3~3D@NE34DG[389Fig. 4a-d. Neocognitron with adaptation, a Response of <strong>the</strong> cells <strong>in</strong>Ucl, Uc2, and Uca to each of <strong>the</strong> four <strong>in</strong>put patterns. Note that <strong>the</strong>number<strong>in</strong>g of <strong>the</strong> cell planes <strong>in</strong> Uc, corresponds to <strong>the</strong> featurenumber<strong>in</strong>g <strong>in</strong> b; b features extracted <strong>by</strong> <strong>the</strong> S-cells <strong>in</strong> <strong>the</strong> first level(note that <strong>the</strong>se are <strong>the</strong> same features as <strong>in</strong> Fig. 3b); e connect<strong>in</strong>gareas between S-layer 2 (vertical) and C-layer 1 (horizontal);d connect<strong>in</strong>g areas between S-layer 3 (vertical) and C-layer 2 (horizontal).In c and d only high connection strengths are shown. Dur<strong>in</strong>glearn<strong>in</strong>g, extra cell-planes were created <strong>in</strong> <strong>the</strong> second and third layer.They are not shown here because <strong>the</strong>y never become activated <strong>in</strong> <strong>the</strong>f<strong>in</strong>al networkIn <strong>the</strong> orig<strong>in</strong>al <strong>neocognitron</strong>, <strong>the</strong> features with which6 can be d<strong>is</strong>t<strong>in</strong>gu<strong>is</strong>hed from 5 (features 8 and 9) were notstrongly extracted <strong>by</strong> cells <strong>in</strong> <strong>the</strong> second level. In <strong>the</strong>network conta<strong>in</strong><strong>in</strong>g adaptation, feature 8 <strong>is</strong> <strong>the</strong> mostimportant feature for recogniz<strong>in</strong>g 6.Although feature 7 can be used to d<strong>is</strong>t<strong>in</strong>gu<strong>is</strong>h 9from all o<strong>the</strong>r patterns, it was not actually usedfor th<strong>is</strong> purpose <strong>in</strong> <strong>the</strong> orig<strong>in</strong>al <strong>neocognitron</strong>. In <strong>the</strong>modified <strong>neocognitron</strong>, however, <strong>the</strong>re are strong connectionslead<strong>in</strong>g from <strong>the</strong> cell-plane which detects th<strong>is</strong>feature.Feature 1 <strong>is</strong> present <strong>in</strong> all <strong>in</strong>put patterns but elicits <strong>the</strong>strongest response when an 8 <strong>is</strong> presented. The strongresponse to th<strong>is</strong> particular feature can thus be regardedas character<strong>is</strong>tic of 8. In <strong>the</strong> modified <strong>neocognitron</strong>, <strong>the</strong>degree of adaptation for th<strong>is</strong> feature <strong>is</strong> so high that only if8 <strong>is</strong> presented <strong>is</strong> <strong>the</strong> cell-plane activated. Consequently,feature 1 <strong>is</strong> used solely for <strong>the</strong> <strong>recognition</strong> of 8, whereas <strong>in</strong><strong>the</strong> orig<strong>in</strong>al <strong>neocognitron</strong> it had been used <strong>in</strong> <strong>the</strong> <strong>recognition</strong>of all numerals.4.3 Recognition of n<strong>in</strong>e numeralsUs<strong>in</strong>g <strong>the</strong> parameter sett<strong>in</strong>gs described <strong>in</strong> Sects. 4.1 and4.2.2, a larger experiment was carried out <strong>in</strong> which <strong>the</strong>network had to recognize n<strong>in</strong>e different numerals, <strong>in</strong>order of presentation: 1, 8, 5, 6, 9, 3, 2, 7, and 4. The orderof presentation <strong>is</strong> not crucial; here we have more or lessgrouped toge<strong>the</strong>r patterns with common features. The<strong>in</strong>put numerals were presented as follows, aga<strong>in</strong> withoutshift<strong>in</strong>g <strong>the</strong>ir position: only <strong>the</strong> first pattern 30 times, <strong>the</strong>n<strong>the</strong> first two patterns 30 times, <strong>the</strong>n <strong>the</strong> first three patternscyclically, etc., until all numericals had been presented.With <strong>the</strong> parameter sett<strong>in</strong>gs used, <strong>the</strong> orig<strong>in</strong>al<strong>neocognitron</strong> developed only three cells <strong>in</strong> <strong>the</strong> highestlayer that responded specifically to one of <strong>the</strong> <strong>in</strong>putpatterns. Only <strong>the</strong> numerals 1, 8, and 4 could be d<strong>is</strong>t<strong>in</strong>gu<strong>is</strong>hedcorrectly. One cell responded to both 5 and 6,and ano<strong>the</strong>r to 9, 3, 2, and 7. With <strong>the</strong> addition ofadaptation, on <strong>the</strong> o<strong>the</strong>r hand, all numerals were correctlyidentified, i.e., each cell <strong>in</strong> <strong>the</strong> highest layerresponded to only one of <strong>the</strong> <strong>in</strong>put patterns. Also <strong>in</strong>experiments <strong>in</strong> which <strong>the</strong> positions of <strong>the</strong> <strong>in</strong>put patternswere shifted dur<strong>in</strong>g <strong>the</strong> presentation, <strong>the</strong> <strong>neocognitron</strong>with adaptation was found to perform markedly betterthan <strong>the</strong> one without. In <strong>the</strong> orig<strong>in</strong>al <strong>neocognitron</strong> <strong>the</strong>parameters had to be tediously set each time so as to beoptimal for <strong>the</strong> particular task at hand. Our experiencesuggests that <strong>by</strong> <strong>in</strong>corporat<strong>in</strong>g adaptation <strong>in</strong>to <strong>the</strong> network<strong>the</strong> parameter range <strong>is</strong> broadened.5 Conclusions and d<strong>is</strong>cussionThe <strong>neocognitron</strong> has been modified to <strong>in</strong>corporateneuronal adaptation, <strong>by</strong> means of which <strong>the</strong> networksuppresses frequently occurr<strong>in</strong>g features (or comb<strong>in</strong>ationsof features) and enhances relatively <strong>in</strong>frequentfeatures (i.e., d<strong>is</strong>crim<strong>in</strong>at<strong>in</strong>g features) with<strong>in</strong> a set of<strong>in</strong>put patterns. Through adaptation, cells will becomeprogressively less responsive to <strong>the</strong> more frequentfeatures, and, consequently, circuits for detect<strong>in</strong>g <strong>the</strong>mwill not be re<strong>in</strong>forced. Circuits for detect<strong>in</strong>g <strong>the</strong> mostd<strong>is</strong>crim<strong>in</strong>at<strong>in</strong>g features of <strong>the</strong> stimulus develop preferentiallyunder <strong>the</strong>se conditions, as a result of which <strong>the</strong>ability of <strong>the</strong> network to differentiate patterns <strong>is</strong> markedly<strong>improved</strong>.In general, <strong>the</strong> concept of adaptation or habituation<strong>is</strong> not restricted to s<strong>in</strong>gle neurons but could also beapplied to entire neural systems, e.g., it might be used <strong>in</strong>expla<strong>in</strong><strong>in</strong>g phenomena such as selective attention.Adaptation <strong>is</strong> <strong>in</strong>corporated <strong>in</strong> C-cells ra<strong>the</strong>r than <strong>in</strong>S-cells. As S-cells compete with each o<strong>the</strong>r to respond toa particular feature, a weaker response due to adaptationwould make <strong>the</strong>m less competitive. Then, a new S-cellcould develop which extracts <strong>the</strong> same feature as <strong>the</strong>S-cell that became previously adapted to it, s<strong>in</strong>ce <strong>the</strong>adapted cell can no longer <strong>in</strong>hibit <strong>the</strong> development of <strong>the</strong>new one.In <strong>the</strong> case where <strong>the</strong> elementary features detected at<strong>the</strong> first level occur <strong>in</strong> all <strong>the</strong> <strong>in</strong>put patterns (i.e., onlycomb<strong>in</strong>ations of features are d<strong>is</strong>t<strong>in</strong>ctive), adaptation
52should only be moderate <strong>in</strong> <strong>the</strong> first level, so that <strong>the</strong>response <strong>is</strong> not entirely suppressed and <strong>the</strong> succeed<strong>in</strong>glevels can, <strong>by</strong> means of adaptation, enhance those comb<strong>in</strong>ationsof features that are d<strong>is</strong>crim<strong>in</strong>at<strong>in</strong>g.In <strong>the</strong> cognitron, a precursor of <strong>the</strong> <strong>neocognitron</strong>,it was also recognized that frequent features were preferentiallylearned. To improve <strong>the</strong> cognitron's patternselectivity, modifiable feedback connections were<strong>in</strong>troduced (Miyake and Fukushima 1984) so that a cellcan suppress its presynaptic cells <strong>by</strong> <strong>in</strong>hibitory feedbackif it <strong>is</strong> activated <strong>by</strong> a feature already familiar to <strong>the</strong>network. In th<strong>is</strong> way, familiar features fail to elicit a susta<strong>in</strong>edresponse, and only circuits detect<strong>in</strong>g novel featureswill develop. Th<strong>is</strong> modification was shown toimprove pattern selectivity and learn<strong>in</strong>g speed. However,<strong>in</strong> order to achieve th<strong>is</strong> improvement, ra<strong>the</strong>r drasticmodifications were made <strong>in</strong> <strong>the</strong> structure of <strong>the</strong>cognitron, as a new type of <strong>in</strong>hibitory cells with feedbackconnections had to be <strong>in</strong>serted. Additionally, onehad to assume that <strong>the</strong> <strong>in</strong>put patterns were presented <strong>in</strong>a specific way. Fur<strong>the</strong>rmore, <strong>the</strong> response character<strong>is</strong>ticsof all <strong>the</strong> cells <strong>in</strong> <strong>the</strong> network were affected <strong>by</strong> <strong>the</strong>modification. In contrast to th<strong>is</strong> global modification,adaptation need not <strong>in</strong>volve more than a local change <strong>in</strong><strong>the</strong> network.For many neuronal properties it <strong>is</strong> not clear what role<strong>the</strong>y could play <strong>in</strong> <strong>the</strong> <strong>in</strong>formation process<strong>in</strong>g carried out<strong>by</strong> biotic neural networks (Selverston 1988). Model neuronsare <strong>the</strong>refore usually simplified to <strong>the</strong> po<strong>in</strong>t thatonly <strong>the</strong> most basic response character<strong>is</strong>tics are considered(summation of <strong>in</strong>puts, fir<strong>in</strong>g threshold). However,to <strong>in</strong>corporate neuronal properties solely to make <strong>the</strong>mmore "real<strong>is</strong>tic", without study<strong>in</strong>g how <strong>the</strong>y actuallyaffect <strong>the</strong> behavior of <strong>the</strong> network, will not give a better<strong>in</strong>sight <strong>in</strong>to <strong>the</strong> role of each property <strong>in</strong> <strong>the</strong> various taskscarried out <strong>by</strong> biotic neural networks (see also Kohonen1992). In th<strong>is</strong> study, we have focused on <strong>the</strong> possible roleof one such property, namely adaptation, <strong>in</strong> v<strong>is</strong>ual pattern<strong>recognition</strong>. We have shown that add<strong>in</strong>g adaptationto a neural network model for v<strong>is</strong>ual pattern <strong>recognition</strong>improves its d<strong>is</strong>crim<strong>in</strong>atory power.Acknowledgements. We are grateful to M.A. Corner for critically read<strong>in</strong>g<strong>the</strong> manuscript and for correct<strong>in</strong>g <strong>the</strong> Engl<strong>is</strong>h, and to M. Timmermanfor ass<strong>is</strong>tance with <strong>the</strong> figures.AppendixFormulas govern<strong>in</strong>g <strong>the</strong> networkIn th<strong>is</strong> section, <strong>the</strong> formulas from Fukushima andMiyake (1982) relevant for our modification will bedescribed.The output of an S-cell of <strong>the</strong> ktth cell-plane of S-cells<strong>in</strong> <strong>the</strong>/th level <strong>is</strong> given <strong>by</strong>Usl(kt, n) = r;" q~f ]1 + 2..k,_, = I )_,v~s at(kt- l, v, k3" Uc,- l(kl- 1, n + v)x ' 1rl1 + ~ "bt(k;)" Vct- l(n)(6)where q~ [x] = x if x/> 0 and q~ [x] = 0 if x < O; n <strong>is</strong> <strong>the</strong>two-dimensional coord<strong>in</strong>ates <strong>in</strong>dicat<strong>in</strong>g <strong>the</strong> location of <strong>the</strong>cell; al(kt- 1, n, kl) and bz(kl) are <strong>the</strong> strengths of <strong>the</strong> variableexcitatory connection and <strong>in</strong>hibitory connection, respectively;St denotes <strong>the</strong> size of <strong>the</strong> connect<strong>in</strong>g area; K;_ 1 <strong>is</strong> <strong>the</strong>total number of cell-planes of C-cells; and r~ <strong>is</strong> a parametercontroll<strong>in</strong>g <strong>the</strong> <strong>in</strong>tensity of <strong>the</strong> <strong>in</strong>hibition. For <strong>the</strong> <strong>in</strong>putlayer, l = O.The output of an <strong>in</strong>hibitory V-cell <strong>is</strong> given <strong>by</strong>Vc'-l(n)= ~K~, = ', ,~s, ~' ct-l(v)'u~t-l(kt-l'n+(7)where cz- 1 (V) represents <strong>the</strong> strength of <strong>the</strong> fixed excitatoryconnections com<strong>in</strong>g from <strong>the</strong> preced<strong>in</strong>g C-cells. The termVct l(n) has <strong>the</strong> same connect<strong>in</strong>g area, St, as Usl(k;, n).The output of a C-cell of <strong>the</strong> kth cell-plane of C-cells <strong>in</strong><strong>the</strong>/th level <strong>is</strong> given <strong>by</strong>+ -~ ~k =, 2~o, dt(v)'ust(k, n + v)where O [x] = x/(a + x) if x ~> 0 and ~ Ix] = 0 if x < O,be<strong>in</strong>g a parameter determ<strong>in</strong><strong>in</strong>g <strong>the</strong> degree of saturation of<strong>the</strong> output; dz(u) <strong>is</strong> <strong>the</strong> value of <strong>the</strong> fixed excitatory connection;and D; <strong>is</strong> <strong>the</strong> connect<strong>in</strong>g area for Uct(k, n).The variable connections al(kt_l, v, k;) and bz(kt) areupdated as follows. Let cell Ust(k, ii) be selected as an MOC.The connections to th<strong>is</strong> cell and to all <strong>the</strong> cells <strong>in</strong> <strong>the</strong> samecell-plane are re<strong>in</strong>forced:Aat(k;- 1, V, kl) = qt" ct- 1 (v)" Uc;- 1 (kl- 1, fi + v) (9)Abt(l~;) = q," Vc;- 1 (fi) (10)where q~ <strong>is</strong> a parameter determ<strong>in</strong><strong>in</strong>g <strong>the</strong> speed of re<strong>in</strong>forcement.In Fukushima and Miyake (1982), a and b <strong>in</strong>creasewithout bound. In our implementation, <strong>the</strong> connectionstrength <strong>is</strong> saturated:m-1a~a'- + Aa (11)mm-1b ~ b" + Ab (12)mwhere m gives <strong>the</strong> duration of <strong>the</strong> memory of <strong>the</strong> connectionsa and b expressed <strong>in</strong> learn<strong>in</strong>g cycles. The <strong>in</strong>itialstrength of <strong>the</strong> variable <strong>in</strong>hibitory connections are set tozero. All <strong>the</strong> variable excitatory <strong>in</strong>put connections to anS-cell get <strong>in</strong>itially a very small value, which <strong>is</strong> proportionalto <strong>the</strong> strength of <strong>the</strong> correspond<strong>in</strong>g fixed <strong>in</strong>put connectionto <strong>the</strong> V-cell.ReferencesBarnard E, Casasent D (1990) Shift <strong>in</strong>variance and <strong>the</strong> <strong>neocognitron</strong>.Neural Networks 3:403-410(8)
53Carpenter G, Grossberg S (1987) Art2: stable self-organization of pattern<strong>recognition</strong> codes for analog <strong>in</strong>put patterns. Appl Opt 26:4919-4930Connors B, Gutnick M J, Pr<strong>in</strong>ce DA (1982) Electrophysiological propertiesof neocortical neurons <strong>in</strong> vitro. J Neurophysiol 48:1302-1320De Boer RJ (1989) Clonal selection versus idiotypic network models of <strong>the</strong>immune system. Thes<strong>is</strong>, University of Utrecht, The Ne<strong>the</strong>rlands, p 6Desimone R (1992) The physiology of memory: record<strong>in</strong>gs of th<strong>in</strong>gs past.Science 258:245-246.Fukushima K (1980) Neocognitron: a self-organ<strong>is</strong><strong>in</strong>g neural network fora mechan<strong>is</strong>m of pattern <strong>recognition</strong> unaffected <strong>by</strong> shift <strong>in</strong> position.Biol Cybern 36:193-202.Fukushima K (1988) Neocognitron: a hierarchical neural network capableof v<strong>is</strong>ual pattern <strong>recognition</strong>. Neural Networks 1:119-130Fukushima K (1989) Analys<strong>is</strong> of <strong>the</strong> process of v<strong>is</strong>ual pattern <strong>recognition</strong><strong>by</strong> <strong>the</strong> <strong>neocognitron</strong>. Neural Networks 2:413-420Fukushima K, lmagawa T (1993) Recognition and segmentation of connectedcharacters with selective attention. Neural Networks 6:33-41Fukushima K, Myake S (1982) Neocognitron: a new algorithm for pattern<strong>recognition</strong> tolerant of deformations and shifts <strong>in</strong> position. <strong>Pattern</strong>Recognition 15:455-469Hubel DH, Wiesel TN (1962) Receptive fields, b<strong>in</strong>ocular <strong>in</strong>teraction andfunctional architecture <strong>in</strong> cat's v<strong>is</strong>ual cortex. J Physiol (Lond)160:106-154Hubel DH, Wiesel TN (1965) Receptive fields and functional architecture<strong>in</strong> two nonstriate v<strong>is</strong>ual area (18 and 19) of <strong>the</strong> cat. J Neurophysiol28:229-289Kohonen T (1984) Self-organization and associative memory. Spr<strong>in</strong>ger,New York Berl<strong>in</strong> HeidelbergKohonen T (1992) Artifical neural networks: models, paradigmsor methods? In: Aleksander I, Taylor J (eds) Artificial neuralnetworks, vol 1. Proceed<strong>in</strong>gs of ICANN-92. Elsevier, Amsterdam,pp 3-10Meech RW (1978) Calcium-dependent potassium activation <strong>in</strong> nervoust<strong>is</strong>sues. Annu Rev Biophys Bioeng 7:1-18Menon MM, He<strong>in</strong>emann KG (1988) Classification of patterns us<strong>in</strong>g a selforganiz<strong>in</strong>gneural network. Neural Networks 1:201-215Miller EK, Li L, Desimone R (1991) A neural mechan<strong>is</strong>m for work<strong>in</strong>g and<strong>recognition</strong> memory <strong>in</strong> <strong>in</strong>ferior temporal cortex. Science254:1377-1379Miyake S, Fukushima K (1984) A neural network model for <strong>the</strong> mechan<strong>is</strong>mof feature-extraction, a self-organiz<strong>in</strong>g network with feedback<strong>in</strong>hibition. Biol Cybern 50:377-384Reilly DL, Cooper LN, Elbaum C (1982) A neural model for categorylearn<strong>in</strong>g. Biol Cybern 45:35-41Rolls ET, Miyashita Y, Cahusac PMB, Kesner RP, Niki H, FeigenbaumJD, Bach L (1989) Hippocampal neurons <strong>in</strong> <strong>the</strong> monkey with activityrelated to <strong>the</strong> palce <strong>in</strong> which a stimulus <strong>is</strong> shown. J. Neurosci.9:1835-1845Selverston AI (1988) A consideration of <strong>in</strong>vertebrate central pattern generatorsas computational data bases. Neural Networks 1:109-117Traub RD (1982) Simulation of <strong>in</strong>tr<strong>in</strong>sic burst<strong>in</strong>g <strong>in</strong> CA3 hippocampalneurons.Neuroscience 7:1233-1242Trot<strong>in</strong> A, Darbel N, Jutand F (1991) Acquired structure, adapted parameters:modifications of <strong>the</strong> Neocognitron. In: Aleksander I, TaylorJ (eds) Artificial neural networks, vol 2. Proceed<strong>in</strong>gs of ICANN-92.Elsevier, Amsterdam, pp 1107-1110