[Bug] Deepseek V3 on GB200 loss nan
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
> **Describe the bug**
>
> When I refer to this [script](https://github.com/NVIDIA/Megatron-LM/blob/dev/examples/moe_recipes/deepseek_v3/gb200/mxfp8_256GPU_TP1PP4EP64.yaml) to pretrain DeepSeek-V3, I found that the gradient norm becomes NaN with commit [d64ba4c](https://github.com/NVIDIA/Megatron-LM/commit/d64ba4ccb1e3e878c15171c9cc58d5d3b46bf4d5).
>
> I also tested a reduced-layer version of DeepSeek-V3 with commit [d64ba4c](https://github.com/NVIDIA/Megatron-LM/commit/d64ba4ccb1e3e878c15171c9cc58d5d3b46bf4d5). In that case, the gradient norm is unusually high, and the training loss is not NaN but is significantly higher than expected. Brown line is runned with commit d64ba4ccb1e3e878c15171c9cc58d5d3b46bf4d5. Red line is normal and runned with commit 1b47bc02f7d364fb123535f84ad89c9e110ef8bc.
>
> 
> **Steps/Code to reproduce bug**
>
> Please use this script to reproduce.
>
> [gb200_2node.sh](https://github.com/user-attachments/files/29969514/gb200_2node.sh) Reproduce the bug with commit [d64ba4c](https://github.com/NVIDIA/Megatron-LM/commit/d64ba4ccb1e3e878c15171c9cc58d5d3b46bf4d5) in dev branch.
>
> **Expected behavior**
>
> Loss decreases normally.
Contributor guide
Assessment
This issue has not been assessed yet.