MTRNNを用いた階層的言語構造の創発 - 奥乃研究室 - 京都大学

RJ2002IO : 30 node, τ = 2a(t+1) b(t +1) … z(t+1) …… …Cf : 40 node, τ = 5…Cs : 11 node, τ = 70Csc : 6 node……I IO , I Cf , I Cs , I Csc : (I Csc ⊂ I Cs )IO Node Numbera(t)30 (?)25 (y)20 (t)15 (o)10 (j)5 (e)b(t)………z(t) ……Fig.1 MTRNN p u n c h t h e s m a l l b a l l .5 10 15 20 25 30 35 40 45StepFig.2 : “punch the small ball.”……I all : I IO ∪ I Cf ∪ I Csu t,i : t i b i : i Csc 0,i : MTRNN τ i : i w ij : j i w ij = 0···(i∈I IO ∧j∈I Cs )∨(i∈I Cs ∧j∈I IO )x t,j : t j MTRNN Back Propagation Through Time(BPTT) [11] . (w ij ), (b i ), (Ccs 0,i ) MTRNN A Z , , , , 4 30 (IO) , , Fast Context(Cf,40 ) Slow Context(Cs, 11 ) 2 (Fig. 1)IO, Cf, Cs (τ) , . Cs 6 Controlling SlowContext(Csc) Csc , IO (Fig. 2). IO t i (y t,i ) ⎧exp(u t,i + b i )∑···(i∈I IO )⎪⎨ exp(u t,j + b j )y t,i = j∈I IO(1)1⎪⎩···(i/∈I IO )1 + exp(−(u t,i + b i ))⎧0 ···(t=0∧i/∈I Csc )⎪⎨Cscu t,i = 0,i ···(t=0∧i∈I Csc )⎪⎩(1 − 1 )u t−1,i + 1 [ ∑ ] (2)w ij x t,j ···(o/w)τ i τ ij∈I allx t,j = y t−1,j ···(t≥1) (3)w (n+1)ijb (n+1)i= w (n)ij= w (n)ij= b (n)i= b (n)i− η ∂E∂w ij− η ∑τ itx t,j∂E(4)∂u t,i− β ∂E∂b i− β ∑ ∂E(5)∂ut t,iCcs (n+1)0,i = Ccs (n)0,i − α ∂E∂Ccs 0,i= Ccs (n)0,i − α ∂E∂u 0,i···(i∈I Csc ) (6)E = ∑ ∑ ( y∗ )yt,i ∗ t,i· log(7)ytt,ii∈I⎧IOy t,i − yt,i ∗ + (1 − 1 ∂E) ···(i∈I IO )τ i ∂u t+1,i∂E⎪⎨= y∂u t,i (1 − y t,i ) ∑ w ki ∂E(8)t,i τ k ∂u t+1,kk∈I all⎪⎩ +(1 − 1 ∂E)τ i∂u t+1,i···(o/w)n : E : yt,i ∗ : t i η, β, α : BPTT IO (x t,j ) (9) x t,j = (1 − r) × y t−1,j + r × yt−1,j ∗ ···(t≥1∧j∈I IO)(9)r : (0 ≤ r ≤ 1)28 日 本 ロッ 術 講 (200 年 22 日 〜24 日