pdfcoffee

Word Embeddings
We will also convert our labels to categorical or one-hot encoding format, because
the loss function we would like to choose (categorical cross-entropy) expects to see
the labels in that format:
# labels
NUM_CLASSES = 2
cat_labels = tf.keras.utils.to_categorical(
labels, num_classes=NUM_CLASSES)
The tokenizer allows access to the vocabulary created through the word_index
attribute, which is basically a dictionary of vocabulary words to their index positions
in the vocabulary. We also build the reverse index that enables us to go from index
position to the word itself. In addition, we create entries for the PAD character:
# vocabulary
word2idx = tokenizer.word_index
idx2word = {v:k for k, v in word2idx.items()}
word2idx["PAD"] = 0
idx2word[0] = "PAD"
vocab_size = len(word2idx)
print("vocab size: {:d}".format(vocab_size))
Finally, we create the dataset object that our network will work with. The dataset
object allows us to set up some properties, such as the batch size, declaratively. Here,
we build up a dataset from our padded sequence of integers and categorical labels,
shuffle the data, and split it into training, validation, and test sets. Finally, we set the
batch size for each of the three datasets:
# dataset
dataset = tf.data.Dataset.from_tensor_slices(
(text_sequences, cat_labels))
dataset = dataset.shuffle(10000)
test_size = num_records // 4
val_size = (num_records - test_size) // 10
test_dataset = dataset.take(test_size)
val_dataset = dataset.skip(test_size).take(val_size)
train_dataset = dataset.skip(test_size + val_size)
BATCH_SIZE = 128
test_dataset = test_dataset.batch(BATCH_SIZE, drop_remainder=True)
val_dataset = val_dataset.batch(BATCH_SIZE, drop_remainder=True)
train_dataset = train_dataset.batch(BATCH_SIZE, drop_remainder=True)
[ 246 ]