ContinualAI / ContinualAI/avalanche

Error on unpickling TextLogger after deleting directory where text log was saved.

Open
#1,661 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.1k
Forks
321
PR merge metrics
No merged PRs in 30d

Description

🐛 **Describe the bug**
Summary: When reloading a strategy checkpoint using `maybe_load_checkpoint`, the loggers for that strategy are unpickled, which calls `TextLogger._fobj_deserialize`. If the strategy saved text logs to a directory that no longer exists (for example in a temporary directory), then when loading the checkpoint you get `FileNotFoundError` for any file definition of the form `"path:"`.

🐜 **To Reproduce**
```
os.makedirs("logs", exist_ok=True)
log_fname = "logs/{}.log".format(datetime.now().strftime('%Y-%m-%d_%H-%M-%S'))

text_logger = TextLogger(file=open(log_fname, 'w'))
eval_plugin = EvaluationPlugin(
loggers=[text_logger]
)

strategy = Naive(
model=model,
optimizer=optimizer,
criterion=criterion,
evaluator=eval_plugin,
)

save_checkpoint(
strategy,
"checkpoint.pt"
)

shutil.rmtree("logs")

strategy, _ = maybe_load_checkpoint(
strategy,
"checkpoint.pt"
)
```

🐝 **Expected behavior**
The log directory and file should be created. This is special pickling behavior from avalanche, hence ideally it should either handle all the pickling cases properly or not pickle the text logger.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.