Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Continuous Bag-of-Words (CBoW)
In these models, the context is given by the surrounding words, both before and
after the blank. That way, it becomes much easier to predict the word that best
fills in the blank. Let’s say we’re trying to fill in the following blank:
Figure 11.9 - Filling the [BLANK] at the end
That’s what a trigram model would have to work with. It doesn’t look good—the
possibilities are endless. Now, consider the same sentence, this time containing the
words that follow the blank.
Figure 11.10 - Filling the [BLANK] in the center
Well, that’s easy: The blank is "dog."
"Cool, but what does the bag-of-words have to do with it?"
It is a bag-of-words because it sums up (or averages) the vectors of the context
words ("the," "small," "is," and "barking") and uses it to predict the central word.
"Why is it continuous? What does it even mean?"
It means the vectors are not one-hot-encoded anymore and have continuous values
instead. The vector of continuous values that represents a given word is a called
word embedding.
Word Embeddings
"How do we find the values that best represent each word?"
We need to train a model to learn them. This model is called…
Word2Vec
Word2Vec was proposed by Mikolov, T. et al. in their 2013 paper, "Efficient
Estimation of Word Representations in Vector Space," [180] and it included two
Word Embeddings | 919