modelscope / modelscope/ms-swift

Qwen3-Next-80B-A3B训练迭代1817个Step卡住,然而进程均正常进行。

Open
#8,081 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Checklist / 检查清单
  • I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。
Bug Description / Bug 描述

基于swift:3.11.1对Qwen3-Next-80B-A3B进行全参数量训练。
4台A800 80G,共32卡训练。训练脚本如下:
megatron sft
--model Qwen/Qwen3-Next-80B-A3B-Thinking
--load_safetensors true
--save_safetensors true
--dataset 'XXXXX'
--loss_scale ignore_empty_think
--tensor_model_parallel_size 1
--pipeline_model_parallel_size 8
--expert_model_parallel_size 4
--sequence_parallel true
--micro_batch_size 1
--global_batch_size 64
--moe_aux_loss_coeff 0.01
--moe_grouped_gemm true
--moe_shared_expert_overlap true
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 1
--max_epochs 3
--finetune true
--cross_entropy_loss_fusion true
--lr 1e-4
--min_lr 1e-10
--save megatron_output/XXXXX
--save_interval 1500
--max_length 10240
--num_workers 128
--dataset_num_proc 128
--no_save_optim true
--no_save_rng true
--sequence_parallel true
--use_flash_attn true
--packing true
--log_interval 1
--moe_expert_capacity_factor 1
训练了2次,都会在相同的1817个step卡住,表现为logging.json已经不更新了,但显卡占用和进程依旧存在,且观察tmux端没有异常。
已经检查过输入数据均为标准sharegpt格式。

想问下是否有可能是OOM。但之前有遇到过OOM是直接Error,其中一台机器的终端直接挂了。但现在情况是依旧在运行。是否有什么可行的排查方向。

How to Reproduce / 如何复现
Additional Information / 补充信息

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported training command and the run's logging.json around step 1817; compare the two reproductions and collect process, GPU, and distributed-worker logs when updates stop. Check whether the hang is reproducible with the stated Qwen3-Next configuration, then document a confirmed cause or the missing diagnostics needed to isolate it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.