pdfcoffee

Chapter 8
loss += loss_fn(decoder_out[:, t], decoder_pred_t)
variables = (encoder.trainable_variables +
decoder.trainable_variables)
gradients = tape.gradient(loss, variables)
optimizer.apply_gradients(zip(gradients, variables))
return loss / decoder_out.shape[1]
The predict() and evaluate() methods also have similar changes, since they also
implement the new data flow on the decoder side that involves an extra encoder_
out parameter and an extra context return value.
We trained two versions of the seq2seq network with attention, once with
additive (Bahdanau) attention, and once with multiplicative (Luong) attention.
Both networks were trained for 50 epochs instead of 250. However, in both cases,
translations were produced with quality similar to that obtained from the seq2seq
network without attention trained for 250 epochs. The training losses at the end of
training for the seq2seq networks with either attention mechanism were marginally
lower, and the BLEU scores on the test sets were slightly higher, compared with the
seq2seq network without attention:
Network Description
Ending Loss
Ending BLEU score
(training set)
(test set)
seq2seq without attention, trained for 250 epochs 0.0967 4.869e-02
seq2seq with additive attention, trained for 30
epochs
seq2seq with multiplicative attention, trained for
30 epochs
0.0893 5.508e-02
0.0706 5.563e-02
Here are some examples of the translations produced from the two networks.
Epoch numbers and the type of attention used are mentioned with each example.
Notice that even when the translations are not 100% the same as the labels, many
of them are valid translations of the original:
Attention
Type
Epoch-# English French (label) French (predicted)
Bahdanau 20 your cat is fat. ton chat est gras. ton chat est mouille.
25 i had to go back. il m a fallu
retourner.
il me faut partir.
[ 335 ]