pdfcoffee

Neural Network Foundations with TensorFlow 2.0
Summarizing experiments run for recognizing
handwritten charts
So, let's summarize: with five different variants, we were able to improve our
performance from 90.71% to 97.82%. First, we defined a simple layer network
in TensorFlow 2.0. Then, we improved the performance by adding some hidden
layers. After that, we improved the performance on the test set by adding a few
random dropouts in our network, and then by experimenting with different types
of optimizers:
model/accuracy training validation test
simple 89.96% 90.70% 90.71%
2 hidden(128) 90.81% 91.40% 91.18%
dropout(30%) 91.70% 94.42% 94.15% (200 epochs)
RMSProp 97.43% 97.62% 97.64% (10 epochs)
Adam 98.94% 97.89% 97.82% (10 epochs)
However, the next two experiments (not shown in the preceding table) were not
providing significant improvements. Increasing the number of internal neurons
creates more complex models and requires more expensive computations, but
it provides only marginal gains. We have the same experience if we increase
the number of training epochs. A final experiment consisted of changing the BATCH_
SIZE for our optimizer. This also provided marginal results.
Regularization
In this section, we will review a few best practices for improving the training phase.
In particular, regularization and batch normalization will be discussed.
Adopting regularization to avoid overfitting
Intuitively, a good machine learning model should achieve a low error rate on
training data. Mathematically this is equivalent to minimizing the loss function
on the training data given the model:
min: {loss(Training Data | Model)}
[ 36 ]