Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

That’s also a possibility; in this case, it is like resuming training for the hidden
layers, while learning from scratch for the output layer. You’d probably have to
have more data to pull this off, though.
"Could I have changed the whole classifier instead of just the output
layer?"
Sure thing! It would be possible to have a different architecture for the classifier
part, as long as it takes the 9,216 input features produced by the first part of
AlexNet, and outputs as many logits as necessary for the task at hand. In this case,
the whole classifier would be learning from scratch, and you’d need even more
data to pull it off.
The more layers you unfreeze or replace, the more data you’ll
need to fine-tune the model.
We’re sticking with the simplest approach here; that is, replacing the output layer
only.
Technically speaking, we’re only fine-tuning a model if we do not
freeze pre-trained weights; that is, the whole model will be
(slightly) updated. Since we are freezing everything but the last
layer, we are actually using the pre-trained model for feature
extraction only.
"What if I use a different model? Which layer should I replace then?"
The table below covers some of the most common models you may use for transfer
learning. It lists the expected size of the input images, the classifier layer to be
replaced, and the appropriate replacement, given the number of classes for the
task at hand (three in our case):
Model Size Classifier Layer(s) Replacement Layer(s)
AlexNet 224 model.classifier[6] nn.Linear(4096,num_classes)
VGG 224 model.classifier[6] nn.Linear(4096,num_classes)
InceptionV3 299 model.fc nn.Linear(2048,num_classes)
model.AuxLogits.fc
nn.Linear(768,num_classes)
512 | Chapter 7: Transfer Learning