ContinualAI / ContinualAI/avalanche
Bug in EarlyStoppingPlugin
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 321
- PR merge metrics
- No merged PRs in 30d
Description
I discovered this bug during an experiment using an IncrementalClassifier as last layer of an MLP.
The EarlyStoppingPlugin is used to re-load the best model, using a validation stream and an eval phase performed every training epoch to check the model performance and avoid overfitting.
The problem raises when the best accuracy obtained for the current experience is less than the best accuracy obtained at some previous experience. In fact, using an IncrementalClassifier, the active units are updated when the model sees new classes (e.g., in the current experience), but then an old state_dict is re-loaded because the best accuracy was better and the active units are not more coherent with the classes seen so far.
Example:
experience0 has examples of classes [0, 1]
experience1 has examples of classes [2, 3]
experience2 has examples of classes [4, 5]
experience3 has examples of classes [6, 7]
At the beginning the IncrementalClassifier starts with 0 output neurons and 0 active units. Then the evalution at the beginning starts and the model adapts to all the classes in the validation stream. Now the model has 18 output neurons, with 0 active units, because it adapted during the evaluation phase.
At the first experience the model sees classes [0, 1], so the active units become [0, 1]. Then it sees classes [2, 3] and the active units become [0, 1, 2, 3]. Then it sees classes [4, 5], so the active units become [0, 1, 2, 3, 4, 5]. At this point in the eval phase the accuracy does not increase, so the model is re-loaded as it was when obtained the best accuracy (e.g., in the previous experience), but there the active units were just [0, 1, 2, 3].
In the next experience it sees classes [6, 7], so the active units are now [0, 1, 2, 3, 6, 7], that are not coherent with the classes seen during the training.
A possible solution could be to reset the best accuracy of the EarlyStoppingPlugin after every experience, in order to reload the model adapted to the current experience (or at least adapted to the actual number of classes seen so far, even if the state_dict was saved at a previous experience).
Contributor guide
Assessment
This issue has not been assessed yet.