ContinualAI / ContinualAI/avalanche
Error on unpickling TextLogger after deleting directory where text log was saved.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 321
- PR merge metrics
- No merged PRs in 30d
Description
🐛 **Describe the bug**
Summary: When reloading a strategy checkpoint using `maybe_load_checkpoint`, the loggers for that strategy are unpickled, which calls `TextLogger._fobj_deserialize`. If the strategy saved text logs to a directory that no longer exists (for example in a temporary directory), then when loading the checkpoint you get `FileNotFoundError` for any file definition of the form `"path:"`.
🐜 **To Reproduce**
```
os.makedirs("logs", exist_ok=True)
log_fname = "logs/{}.log".format(datetime.now().strftime('%Y-%m-%d_%H-%M-%S'))
text_logger = TextLogger(file=open(log_fname, 'w'))
eval_plugin = EvaluationPlugin(
loggers=[text_logger]
)
strategy = Naive(
model=model,
optimizer=optimizer,
criterion=criterion,
evaluator=eval_plugin,
)
save_checkpoint(
strategy,
"checkpoint.pt"
)
shutil.rmtree("logs")
strategy, _ = maybe_load_checkpoint(
strategy,
"checkpoint.pt"
)
```
🐝 **Expected behavior**
The log directory and file should be created. This is special pickling behavior from avalanche, hence ideally it should either handle all the pickling cases properly or not pickle the text logger.
Contributor guide
Assessment
This issue has not been assessed yet.