modelscope / modelscope/ms-swift
Qwen3-Next-80B-A3B训练迭代1817个Step卡住,然而进程均正常进行。
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。
Bug Description / Bug 描述
基于swift:3.11.1对Qwen3-Next-80B-A3B进行全参数量训练。
4台A800 80G,共32卡训练。训练脚本如下:
megatron sft
--model Qwen/Qwen3-Next-80B-A3B-Thinking
--load_safetensors true
--save_safetensors true
--dataset 'XXXXX'
--loss_scale ignore_empty_think
--tensor_model_parallel_size 1
--pipeline_model_parallel_size 8
--expert_model_parallel_size 4
--sequence_parallel true
--micro_batch_size 1
--global_batch_size 64
--moe_aux_loss_coeff 0.01
--moe_grouped_gemm true
--moe_shared_expert_overlap true
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 1
--max_epochs 3
--finetune true
--cross_entropy_loss_fusion true
--lr 1e-4
--min_lr 1e-10
--save megatron_output/XXXXX
--save_interval 1500
--max_length 10240
--num_workers 128
--dataset_num_proc 128
--no_save_optim true
--no_save_rng true
--sequence_parallel true
--use_flash_attn true
--packing true
--log_interval 1
--moe_expert_capacity_factor 1
训练了2次,都会在相同的1817个step卡住,表现为logging.json已经不更新了,但显卡占用和进程依旧存在,且观察tmux端没有异常。
已经检查过输入数据均为标准sharegpt格式。
想问下是否有可能是OOM。但之前有遇到过OOM是直接Error,其中一台机器的终端直接挂了。但现在情况是依旧在运行。是否有什么可行的排查方向。
How to Reproduce / 如何复现
Additional Information / 补充信息
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported training command and the run's logging.json around step 1817; compare the two reproductions and collect process, GPU, and distributed-worker logs when updates stop. Check whether the hang is reproducible with the stated Qwen3-Next configuration, then document a confirmed cause or the missing diagnostics needed to isolate it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100