Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

"lion" or "Dorothy" in the sentence, it’s likely from The Wonderful Wizard of Oz.
Cool, right? Looks like word embeddings are the best invention since sliced bread!
That would indeed be the case if only actual languages were straightforward and
organized—unfortunately, they are nothing like that.
Let me show you two sentences from Alice’s Adventures in Wonderland (the
highlights are mine):
• "The Hatter was the first to break the silence. `What day of the month is it?' he said,
turning to Alice: he had taken his watch out of his pocket, and was looking at it
uneasily, shaking it every now and then, and holding it to his ear."
• "Alice thought this a very curious thing, and she went nearer to watch them, and just
as she came up to them she heard one of them say, `Look out now, Five! Don’t go
splashing paint over me like that!"
In the first sentence, the word "watch" is a noun and it refers to the object the
Hatter had taken out of his pocket. In the second sentence, "watch" is a verb and it
refers to what Alice is doing. Clearly, two very different meanings for the same
word.
But, if we look the "watch" token up in our vocabulary, we’ll always retrieve the
same values from the word embeddings, regardless of the actual meaning of the
word in a sentence.
Can we do better? Of course!
Contextual Word Embeddings
If a single token is not enough, why not take the whole sentence, right? Instead of
taking a word by itself, we can take its context too in order to compute the vector
that best represents a word. That was the whole point of word embeddings:
finding numerical representation for words (or tokens).
"That’s great, but it seems impractical."
You’re absolutely right! Trying to build a lookup table for every possible
combination of word and context is probably not such a great idea—that’s why
contextual word embeddings won’t come from a lookup table but from the
outputs of a model instead.
948 | Chapter 11: Down the Yellow Brick Rabbit Hole