NVIDIA / NVIDIA/Megatron-LM

[main] reduce_scatter_with_fp32_accumulation has a bug and cannot run normally.

Open
#5,177 2 comments 0 reactions 1 assignee Claimed by @deepakn94 View on GitHub
bug community-request waiting-on-maintainers
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**
reduce_scatter_with_fp32_accumulation has a bug and cannot run normally.

**Steps/Code to reproduce bug**
branch: main
commit: b0eb9143c
- pytorch: 2.6.0
- transformer_engine: 2.7.0+fedd9ddc
- cuda driver version: 12.8
- cuda toolkit version: 12.1

shell:
torchrun --nproc_per_node 2 --nnodes 1 --node_rank 0 --master_addr gpu1 --master_port 20070 pretrain_gpt.py --use-mcore-models --num-attention-heads 32 --group-query-attention --num-query-groups 8 --kv-channels 128 --seq-length 1024 --max-position-embeddings 1024 --position-embedding-type rope --rotary-base 1000000 --rotary-percent 1.0 --attention-dropout 0.0 --hidden-dropout 0.0 --swiglu --init-method-std 0.0134 --attention-backend flash --apply-layernorm-1p --untie-embeddings-and-output-weights --disable-bias-linear --no-rope-fusion --normalization RMSNorm --no-persist-layer-norm --moe-grouped-gemm --num-layers 8 --hidden-size 4 --ffn-hidden-size 4 --num-experts 8 --moe-router-topk 1 --moe-ffn-hidden-size 4 --micro-batch-size 1 --global-batch-size 4 --train-iters 4 --lr 0.00015 --min-lr 0.00001 --decoupled-lr 5.0e-4 --decoupled-min-lr 4.5e-5 --lr-decay-style cosine --clip-grad 1.0 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --bf16 --grad-reduce-in-bf16 --cross-entropy-loss-fusion --manual-gc --empty-unused-memory-level 1 --exit-duration-in-mins 235 --no-gradient-accumulation-fusion --moe-token-dispatcher-type alltoall --moe-router-pre-softmax --logging-level 20 --ddp-reduce-scatter-with-fp32-accumulation --use-distributed-optimizer --overlap-grad-reduce --overlap-param-gather --tensor-model-parallel-size 1 --context-parallel-size 1 --pipeline-model-parallel-size 1 --sequence-parallel --expert-model-parallel-size 2 --expert-tensor-parallel-size 1 --data-path /home/u2025110710/datasets/mcore/wiki_text_document --tokenizer-type HuggingFaceTokenizer --tokenizer-model /home/u2025110710/models/qwen3-30b-a3b --data-cache-path /home/u2025110710/repos/megatron/benchmark_cache_llama3_8b_fp8 --split '99,1,0' --no-create-attention-mask-in-dataloader --no-mmap-bin-files --num-workers 1 --vocab-size 151936 --eval-iters 0 --eval-interval 2147483647 --log-interval 1 --log-throughput --profile --profile-step-start 5 --profile-step-end 6 --profile-ranks 0 --tensorboard-dir tensorboard_logs/llama3_8b/nopp

Image

**Expected behavior**
normally run.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.