ContinualAI / ContinualAI/avalanche

Checkpointing issues in 0.4.0

Open
#1,549 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.1k
Forks
321
PR merge metrics
No merged PRs in 30d

Description

When trying to save a checkpoint the, `torch.save(checkpoint_data, fname, pickle_module=dill)` call in `save_checkpoint()` function fails. It complains about a recursive definition when calling `torch.save()` inside the respective function.

**Additional context:**
I've followed the checkpoint example with the few changes below.
The model is a CNN model (inheriting from `DynamicModule`) with an IncrementalClassifier as the output layer. This is in order to do class-incremental learning. Starts with 2 output classes and adapts in the train-eval loop.
The dataset is MNIST loaded from torchvision (not avalanche) and the benchmark is created with `nc_benchmark` with 10 experiences and `task_labels=False`. Only `ToTensor()` transform is used for the data.
The strategy is `Naive`. I'm using a `ReplayBuffer`, which I try removing it but still fails.
The evaluation plugin has the `accuracy_metrics`, the `forgetting_metrics`, and an interactive logger.
The main loop calls adaptation(), train(), eval() in that order.

**System:** Mac 2023 with M2

**Versions: **
```
avalanche-lib 0.4.0
torch 2.1.1
torchvision 0.16.1
python 3.10.13
```

```shell
/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py:412: PicklingWarning: Cannot locate reference to .
StockPickler.save(self, obj, save_persistent_id)
/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py:412: PicklingWarning: Cannot pickle : typing.ParamSpec has recursive self-references that trigger a RecursionError.
StockPickler.save(self, obj, save_persistent_id)
Traceback (most recent call last):
File "/Users/me/projects/cl/train.py", line 172, in
main(args)
File "/Users/me/projects/cl/train.py", line 144, in main
save_checkpoint(strategy, filename)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/avalanche/training/checkpoint.py", line 88, in save_checkpoint
torch.save(checkpoint_data, fname, pickle_module=dill)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/torch/serialization.py", line 619, in save
_save(obj, opened_zipfile, pickle_module, pickle_protocol, _disable_byteorder_record)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/torch/serialization.py", line 831, in _save
pickler.dump(obj)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 418, in dump
StockPickler.dump(self, obj)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 487, in dump
self.save(obj)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1212, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 972, in save_dict
self._batch_setitems(obj.items())
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
save(v)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 717, in save_reduce
save(state)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1212, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 972, in save_dict
self._batch_setitems(obj.items())
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
save(v)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 687, in save_reduce
save(cls)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1812, in save_type
_save_with_postproc(pickler, (_create_type, (
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1093, in _save_with_postproc
pickler.save_reduce(*reduction, obj=obj)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 692, in save_reduce
save(args)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 902, in save_tuple
save(element)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 887, in save_tuple
save(element)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1812, in save_type
_save_with_postproc(pickler, (_create_type, (
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1107, in _save_with_postproc
pickler._batch_setitems(iter(source.items()))
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
save(v)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1812, in save_type
_save_with_postproc(pickler, (_create_type, (
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1093, in _save_with_postproc
pickler.save_reduce(*reduction, obj=obj)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 692, in save_reduce
save(args)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 902, in save_tuple
save(element)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1212, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 972, in save_dict
self._batch_setitems(obj.items())
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
save(v)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1990, in save_function
StockPickler.save_global(pickler, obj, name=name)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 1100, in save_global
self.save_reduce(getattr, (parent, lastname))
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 692, in save_reduce
save(args)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 887, in save_tuple
save(element)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1849, in save_type
StockPickler.save_global(pickler, obj, name=obj_name)
File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 1076, in save_global
raise PicklingError(
_pickle.PicklingError: Can't pickle : it's not the same object as typing.ParamSpec
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.