pdfcoffee

Recurrent Neural Networks
30 try to find it. tentez de le
trouver.
tentez de le trouver.
Luong 20 that s peculiar. c est etrange. c est deconcertant.
25 tom is athletic. thomas est sportif. tom est sportif.
30 it s dangerous. c est dangereux. c est dangereux.
The full code for the network described here is in the seq2seq_with_attn.py file
in the code folder for this chapter. To run the code from the command line, please
use the following command. You can switch between Bahdanau (additive) or Luong
(multiplicative) attention mechanisms by commenting out one or the other in the
init() method of the Decoder class:
$ python seq2seq_with_attn.py
In the next section, we will describe the next architectural landmark in text
processing using deep neural networks, the transformer network, which combines
ideas from Encoder-Decoder architecture and attention.
Transformer architecture
Even though the transformer architecture is different from recurrent networks,
it uses many ideas that originated in recurrent networks. It represents the next
evolutionary step of deep learning architectures that work with text, and as such,
should be an essential part of your toolbox. The transformer architecture is a variant
of the Encoder-Decoder architecture, where the recurrent layers have been replaced
with Attention layers. The transformer architecture was proposed by Vaswani, et al.
[30], and a reference implementation provided, which we will refer to throughout
this discussion.
Figure 7 shows a seq2seq network with attention and compares it to a transformer
network. The transformer is similar to the seq2seq with Attention model in the
following ways:
1. Both source and target are sequences
2. The output of the last block of the encoder is used as context or thought
vector for computing the Attention model on the decoder
[ 336 ]