The first experiment earns its place
A pretrained model offers a useful starting representation. It does not remove the need for a baseline.
Before tuning the whole network, establish a simple reference. Freeze the feature extractor, train an appropriate classification head, and evaluate the result with a fixed data split and a clear training procedure.
The baseline tells you what the existing representation can do for your task. It also gives later experiments a meaningful point of comparison.
Change one decision at a time
You can then explore a limited adaptation: unfreeze a later block, adjust the learning rate, and keep the comparison controlled. Record what changed and why.
A compact experiment table helps:
| Experiment | Trainable parts | What stays fixed |
|---|---|---|
| Baseline | Classification head | Split, preprocessing, evaluation |
| Selective fine-tuning | Head and a later block | Split, preprocessing, evaluation |
| Broader fine-tuning | Additional blocks | Split, preprocessing, evaluation |
This is a planning framework, not a claim that one configuration will always win. The useful depth can depend on the dataset, its size, and the training choices.
Look for more than the best run
Training can vary. A small difference between two runs may be less informative than it first appears. When the question warrants it, repeat the comparison and report the variation alongside the average.
More trainable parameters also change compute, memory use, and the opportunity to overfit. Those are part of the decision, even when they do not appear in a headline score.
For the research behind this question, see dataset size and fine-tuning depth. For a practical introduction, explore the PyTorch course.
Practical reference
Written for Insights by Ackley Will. About the author · Share a question or observation