NVIDIA / NVIDIA/Megatron-LM

Gather rounds causes corruption in checkpoint load

Open
#3,738 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

After enabling gather_rounds for checkpoint load performance, we routinely have numerical instability in the workload.

Recommend removing this feature since it is under tested and unstable. We've run into this repeatedly during hero runs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.