NVIDIA / NVIDIA/Megatron-LM

[BUG] --async-save mistakenly recorded the time when the model was saved in the log

Open
#1,763 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

🐛 Bug Report

Description
When using synchronous checkpoint saving, the program prints messages like:

save-checkpoint ................................: (39899.66, 39899.95)

after all model weights have been successfully saved.

However, when using the --async-save option, the program prints the same message immediately after the asynchronous save request is issued, even though the checkpoint has not actually finished saving yet.
This behavior is confusing because it gives the impression that the checkpoint has been fully written, while in reality the async save may still be in progress (or even fail later).

Steps to Reproduce

Train a model with --async-save enabled.

Observe the log output when checkpoints are saved.

Compare with the log output when --async-save is disabled.

Expected Behavior
The save-checkpoint ................................: (xxx, xxx) log should only appear after the checkpoint has been fully saved (consistent with synchronous saving), or at least be clearly marked as "async save requested" to avoid confusion.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.