ContinualAI / ContinualAI/avalanche

Bug in EarlyStoppingPlugin

Open
#1,486 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.1k
Forks
321
PR merge metrics
No merged PRs in 30d

Description

I discovered this bug during an experiment using an IncrementalClassifier as last layer of an MLP.
The EarlyStoppingPlugin is used to re-load the best model, using a validation stream and an eval phase performed every training epoch to check the model performance and avoid overfitting.

The problem raises when the best accuracy obtained for the current experience is less than the best accuracy obtained at some previous experience. In fact, using an IncrementalClassifier, the active units are updated when the model sees new classes (e.g., in the current experience), but then an old state_dict is re-loaded because the best accuracy was better and the active units are not more coherent with the classes seen so far.

Example:
experience0 has examples of classes [0, 1]
experience1 has examples of classes [2, 3]
experience2 has examples of classes [4, 5]
experience3 has examples of classes [6, 7]

At the beginning the IncrementalClassifier starts with 0 output neurons and 0 active units. Then the evalution at the beginning starts and the model adapts to all the classes in the validation stream. Now the model has 18 output neurons, with 0 active units, because it adapted during the evaluation phase.
At the first experience the model sees classes [0, 1], so the active units become [0, 1]. Then it sees classes [2, 3] and the active units become [0, 1, 2, 3]. Then it sees classes [4, 5], so the active units become [0, 1, 2, 3, 4, 5]. At this point in the eval phase the accuracy does not increase, so the model is re-loaded as it was when obtained the best accuracy (e.g., in the previous experience), but there the active units were just [0, 1, 2, 3].
In the next experience it sees classes [6, 7], so the active units are now [0, 1, 2, 3, 6, 7], that are not coherent with the classes seen during the training.

A possible solution could be to reset the best accuracy of the EarlyStoppingPlugin after every experience, in order to reload the model adapted to the current experience (or at least adapted to the actual number of classes seen so far, even if the state_dict was saved at a previous experience).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.