ContinualAI / ContinualAI/avalanche
Forgetting measure does not work in online scenarios
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 321
- PR merge metrics
- No merged PRs in 30d
Description
🐛 **Describe the bug**
Forgetting measure fails to update correctly in online scenarios
🐜 **To Reproduce**
Add ExperienceForgetting measure to the evaluation metrics in split_cifar10/online_replay.py, the metric does not fail with an error but is not reported by the loggers.
🐝 **Expected behavior**
The metric should be reported by the loggers once the second task has finished
🦋 **Additional context**
I started investigating and I think the following lines are causing the metric to fail:
In avalanche/evaluation/metrics/forgetting_bw.py GenericExperienceForgetting line 213
```{python}
if self.train_exp_id == self.eval_exp_id:
self.update(
self.eval_exp_id, self.metric_result(strategy), initial=True
)
else:
# update other experiences
# if experience has not been encountered in training
# its value will not be considered in forgetting
self.update(self.eval_exp_id, self.metric_result(strategy))
```
The problem is that since this is online learning, self.train_exp_id at the end of training task 1 is very high (like 950 in the case of split cifar10 with bs 10), and does not correspond to self.eval_exp_id (0 for task 0 etc..). So the forgetting is never computed in initial state and thus never reported.
I think it's not necessary that this metric works when task boundaries are not provided, but it would be good that it works when they are provided. You can assign me to this issue I will try to fix it since it also causes CumulativeAccuracyForgetting (the one I'm trying to implement) to fail.
Contributor guide
Assessment
This issue has not been assessed yet.