pdfcoffee

Chapter 7
class_weight=CLASS_WEIGHTS)
# evaluate against test set
labels, predictions = [], []
for Xtest, Ytest in test_dataset:
Ytest_ = model.predict_on_batch(Xtest)
ytest = np.argmax(Ytest, axis=1)
ytest_ = np.argmax(Ytest_, axis=1)
labels.extend(ytest.tolist())
predictions.extend(ytest.tolist())
print("test accuracy: {:.3f}".format(accuracy_score(labels,
predictions)))
print("confusion matrix")
print(confusion_matrix(labels, predictions))
Running the spam detector
The three scenarios we want to look at are:
• Letting the network learn the embedding for the task
• Starting with a fixed external third party embedding where the embedding
matrix is treated like a vectorizer to transform the sequence of integers into
a sequence of vectors
• Starting with an external third party embedding which is further fine-tuned
to the task during the training
Each scenario can be evaluated setting the value of the mode argument as shown
in the following command:
$ python spam_classifier --mode [scratch|vectorizer|finetune]
The dataset is small and the model is fairly simple. We were able to achieve very
good results (validation set accuracies in the high 90s, and perfect test set accuracy)
with only minimal training (3 epochs). In all three cases, the network achieved
a perfect score, accurately predicting the 1,111 ham messages, as well as the
169 spam cases.
[ 251 ]