NVIDIA / NVIDIA/Megatron-LM

[BUG]loss error when using MLA

Open
#1,445 15 comments 0 reactions 0 assignees View on GitHub
bug community-request module: moe
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**
when using mla by megatron-core 0.10.0,loss was unexpected

**To Reproduce**
use /nvcr.io/nvidia/nemo:25.02.rc1 image start traing

**Additional context**
Add any other context about the problem here.
--num-layers 60 \
--hidden-size 5120 \
--ffn-hidden-size 12288 \
--num-attention-heads 128 \
--seq-length 4096 \
--max-position-embeddings 4096 \
--multi-latent-attention \
--kv-lora-rank 512 \
--q-lora-rank 1536 \
--moe-ffn-hidden-size 1536 \
--moe-router-topk 6 \
--num-experts 160 \
--decoder-first-pipeline-num-layers 2 \
--decoder-last-pipeline-num-layers 2 \
--moe-aux-loss-coeff 1e-2 \
--moe-shared-expert-intermediate-size 3072 \
--moe-layer-freq [0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1] \
--expert-model-parallel-size 8 \
--moe-grouped-gemm \
--moe-router-load-balancing-type aux_loss \
--qk-head-dim 128 \
--qk-pos-emb-head-dim 64 \
--v-head-dim 128 \
--rotary-base 10000 \
--no-rope-fusion \
--no-bias-dropout-fusion \
--no-persist-layer-norm \
--micro-batch-size 1 \

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.