[BUG] --async-save mistakenly recorded the time when the model was saved in the log
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
🐛 Bug Report
Description
When using synchronous checkpoint saving, the program prints messages like:
save-checkpoint ................................: (39899.66, 39899.95)
after all model weights have been successfully saved.
However, when using the --async-save option, the program prints the same message immediately after the asynchronous save request is issued, even though the checkpoint has not actually finished saving yet.
This behavior is confusing because it gives the impression that the checkpoint has been fully written, while in reality the async save may still be in progress (or even fail later).
Steps to Reproduce
Train a model with --async-save enabled.
Observe the log output when checkpoints are saved.
Compare with the log output when --async-save is disabled.
Expected Behavior
The save-checkpoint ................................: (xxx, xxx) log should only appear after the checkpoint has been fully saved (consistent with synchronous saving), or at least be clearly marked as "async save requested" to avoid confusion.
Contributor guide
Assessment
This issue has not been assessed yet.