pdfcoffee

Chapter 7
A more convenient source of sentence embeddings is the Google Universal Sentence
Encoder, available on TensorFlow Hub. There are two flavors of the encoder in
terms of implementation. The first flavor is fast but not so accurate and is based on
the Deep Averaging Network (DAN) proposed by Iyer, et al. [24], which combines
embeddings for words and bigrams and sends it through a fully connected network.
The second flavor is much more accurate but slower, and is based on the encoder
component of the transformer network proposed by Vaswani, et al. [25]. We will
cover the transformer network in more detail in the next chapter.
As with ELMo, the Google Universal Sentence Encoder is currently only available
for non-eager mode of execution, so you can use it offline to generate the vectors
or integrate it into your TensorFlow 1.x style code. One other thing is that the
model does not fit on GPU memory, so you will have to forcibly run it on the
CPU. However, since we are running it in prediction mode this is not a problem.
Here is some code that calls it with two of our example sentences:
import tensorflow as tf
import tensorflow_hub as hub
module_url = "https://tfhub.dev/google/universal-sentence-encoder/2"
tf.compat.v1.disable_eager_execution()
model = hub.Module(module_url)
embeddings = model([
"i like green eggs and ham",
"would you eat them in a box"
])
// turn off GPU
config = tf.ConfigProto(device_count = { "GPU" : 0 }}
with tf.compat.v1.Session(config=config) as sess:
sess.run([
tf.compat.v1.global_variables_initializer(),
tf.compat.v1.tables_initializer()
])
embeddings_value = sess.run(embeddings)
print(embeddings_value.shape)
The output is (2, 512); that is, each sentence is represented by a vector of size (512).
It is important to note that the Google Universal Sentence Encoder can handle any
length of word sequence—so you could legitimately use it to get word embeddings
on one end as well as paragraph embeddings on the other. However, as the sequence
length gets larger, the quality of the embeddings tends to get "diluted".
[ 263 ]