modelscope / modelscope/ms-swift

Packing+多模态数据使用deepspeed微调Qwen3.6-35B-A3B发生数据预处理死锁

Open
#9,701 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Checklist / 检查清单
  • I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。
Bug Description / Bug 描述

NPU910B Qwen3.6-35B-A3B sft死锁,似乎和Packing/多模态数据强相关。

在8*910B上测试
使用的镜像是swift官方提供的swift4.3.0-910B镜像 自己加上了deepspeed的zero3补丁:https://github.com/modelscope/ms-swift/issues/9657

会在这里死锁:

Image Image

键盘中断后调用栈

Image
How to Reproduce / 如何复现

我用的训练命令:
swift sft
--model /tmp/Qwen3.6-35B-A3B
--dataset /tmp/final.jsonl \ #这是我自己制作的一个多模态数据集,Agent数据每个个包含1-3张图片
--split_dataset_ratio 0.5
--torch_dtype bfloat16
--tuner_type lora
--target_modules all-linear
--freeze_vit true
--freeze_aligner true
--lora_rank 8
--lora_alpha 32
--num_train_epochs 16
--packing true \ #这个会导致死锁 CPU占用率为0
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--gradient_accumulation_steps 2
--learning_rate 5e-5
--max_length 40960
--save_steps 100
--eval_steps 100
--save_total_limit 2
--logging_steps 2
--deepspeed zero3_offload
--sequence_parallel_size 8
--padding_free true
--attn_impl flash_attn
--output_dir /tmp/result

具体卡在预处理的这里
https://github.com/modelscope/ms-swift/blob/v4.3.0/swift/dataset/preprocessor/core.py#L353
看不到是卡在哪个数据上了
https://github.com/modelscope/ms-swift/blob/v4.3.0/swift/dataset/preprocessor/core.py#L325
如果把这里的num_proc设置为None,就不会死锁在数据预处理阶段,但是会死锁在Train 0/100

如果关闭packing或者不使用多模态数据,就一切正常

Additional Information / 补充信息

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce with the provided swift sft command, focusing on --packing true with the multimodal dataset. Inspect swift/dataset/preprocessor/core.py around lines 325 and 353, then compare runs with packing disabled, multimodal data removed, or num_proc set to None. Done means preprocessing and training no longer deadlock under the reported configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.