[BUG]loss error when using MLA
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Describe the bug**
when using mla by megatron-core 0.10.0,loss was unexpected
**To Reproduce**
use /nvcr.io/nvidia/nemo:25.02.rc1 image start traing
**Additional context**
Add any other context about the problem here.
--num-layers 60 \
--hidden-size 5120 \
--ffn-hidden-size 12288 \
--num-attention-heads 128 \
--seq-length 4096 \
--max-position-embeddings 4096 \
--multi-latent-attention \
--kv-lora-rank 512 \
--q-lora-rank 1536 \
--moe-ffn-hidden-size 1536 \
--moe-router-topk 6 \
--num-experts 160 \
--decoder-first-pipeline-num-layers 2 \
--decoder-last-pipeline-num-layers 2 \
--moe-aux-loss-coeff 1e-2 \
--moe-shared-expert-intermediate-size 3072 \
--moe-layer-freq [0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1] \
--expert-model-parallel-size 8 \
--moe-grouped-gemm \
--moe-router-load-balancing-type aux_loss \
--qk-head-dim 128 \
--qk-pos-emb-head-dim 64 \
--v-head-dim 128 \
--rotary-base 10000 \
--no-rope-fusion \
--no-bias-dropout-fusion \
--no-persist-layer-norm \
--micro-batch-size 1 \
Contributor guide
Assessment
This issue has not been assessed yet.