pdfcoffee

Chapter 7
Word embeddings are based on the distributional hypothesis, which states that
words that occur in similar contexts tend to have similar meanings. Hence the class
of word embedding-based encodings are also known as distributed representations,
which we will talk about next.
Distributed representations
Distributed representations attempt to capture the meaning of a word by
considering its relations with other words in its context. The idea behind the
distributed hypothesis is captured in this quote from J. R. Firth, a linguist who
first proposed this idea:
"You shall know a word by the company it keeps."
How does this work? By way of example, consider the following pair of sentences:
Paris is the capital of France.
Berlin is the capital of Germany.
Even assuming no knowledge of world geography, the sentence pair implies some
sort of relationship between the entities Paris, France, Berlin, and Germany that
could be represented as:
"Paris" is to "France" as "Berlin" is to "Germany"
Distributed representations are based on the idea that there exists some
transformation φφ such that:
φφ("Paris") − φφ("France") ≈ φφ("Berlin") − φφ("GGGGGGGGGGGGGG")
In other words, a distributed embedding space is one where words that are used
in similar contexts are close to one another. Therefore, similarity between the word
vectors in this space would roughly correspond to the semantic similarity between
the words.
Figure 1 shows a TensorBoard visualization of word embedding of words around the
word "important" in the embedding space. As you can see, the neighbors of the word
tend to be closely related, or interchangeable with the original word.
[ 233 ]