NVIDIA / NVIDIA/Megatron-LM

[Bug] Deepseek V3 on GB200 loss nan

Open
#5,782 3 comments 0 reactions 1 assignee Claimed by @YangFei1990 View on GitHub
bug community-request waiting-on-customer
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

> **Describe the bug**
>
> When I refer to this [script](https://github.com/NVIDIA/Megatron-LM/blob/dev/examples/moe_recipes/deepseek_v3/gb200/mxfp8_256GPU_TP1PP4EP64.yaml) to pretrain DeepSeek-V3, I found that the gradient norm becomes NaN with commit [d64ba4c](https://github.com/NVIDIA/Megatron-LM/commit/d64ba4ccb1e3e878c15171c9cc58d5d3b46bf4d5).
>
> I also tested a reduced-layer version of DeepSeek-V3 with commit [d64ba4c](https://github.com/NVIDIA/Megatron-LM/commit/d64ba4ccb1e3e878c15171c9cc58d5d3b46bf4d5). In that case, the gradient norm is unusually high, and the training loss is not NaN but is significantly higher than expected. Brown line is runned with commit d64ba4ccb1e3e878c15171c9cc58d5d3b46bf4d5. Red line is normal and runned with commit 1b47bc02f7d364fb123535f84ad89c9e110ef8bc.
>
> Image

> **Steps/Code to reproduce bug**
>
> Please use this script to reproduce.
>
> [gb200_2node.sh](https://github.com/user-attachments/files/29969514/gb200_2node.sh) Reproduce the bug with commit [d64ba4c](https://github.com/NVIDIA/Megatron-LM/commit/d64ba4ccb1e3e878c15171c9cc58d5d3b46bf4d5) in dev branch.
>
> **Expected behavior**
>
> Loss decreases normally.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.