Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Wide vs Narrow Attention
This mechanism is known as wide attention: Each attention head gets the
full hidden state and produces a context vector of the same size. This is
totally fine if the number of hidden dimensions is small.
For a larger number of dimensions, though, each attention head will get a
chunk of the affine transformation of the hidden state to work with. This is
a detail of utmost importance: It is not a chunk of the original hidden state,
but of its transformation. For example, say there are 512 dimensions in the
hidden state and we’d like to use eight attention heads: Each attention head
would work with a chunk of 64 dimensions only. This mechanism is known as
narrow attention, and we’ll get back to it in the next chapter.
"Which one should I use?"
On the one hand, wide attention will likely yield better models compared to
using narrow attention on the same number of dimensions. On the other
hand, narrow attention makes it possible to use more dimensions, which
may improve the quality of the model as well. It’s hard to tell you which one
is best overall, but I can tell you that state-of-the-art large Transformer
models use narrow attention. In our much simpler and smaller model,
though, we’re sticking with wide attention.
The multi-headed attention mechanism is usually depicted like this:
Figure 9.22 - Multi-headed attention mechanism
Attention | 737