modelscope / modelscope/ms-swift
Packing+多模态数据使用deepspeed微调Qwen3.6-35B-A3B发生数据预处理死锁
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。
Bug Description / Bug 描述
NPU910B Qwen3.6-35B-A3B sft死锁,似乎和Packing/多模态数据强相关。
在8*910B上测试
使用的镜像是swift官方提供的swift4.3.0-910B镜像 自己加上了deepspeed的zero3补丁:https://github.com/modelscope/ms-swift/issues/9657
会在这里死锁:
键盘中断后调用栈
How to Reproduce / 如何复现
我用的训练命令:
swift sft
--model /tmp/Qwen3.6-35B-A3B
--dataset /tmp/final.jsonl \ #这是我自己制作的一个多模态数据集,Agent数据每个个包含1-3张图片
--split_dataset_ratio 0.5
--torch_dtype bfloat16
--tuner_type lora
--target_modules all-linear
--freeze_vit true
--freeze_aligner true
--lora_rank 8
--lora_alpha 32
--num_train_epochs 16
--packing true \ #这个会导致死锁 CPU占用率为0
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--gradient_accumulation_steps 2
--learning_rate 5e-5
--max_length 40960
--save_steps 100
--eval_steps 100
--save_total_limit 2
--logging_steps 2
--deepspeed zero3_offload
--sequence_parallel_size 8
--padding_free true
--attn_impl flash_attn
--output_dir /tmp/result
具体卡在预处理的这里
https://github.com/modelscope/ms-swift/blob/v4.3.0/swift/dataset/preprocessor/core.py#L353
看不到是卡在哪个数据上了
https://github.com/modelscope/ms-swift/blob/v4.3.0/swift/dataset/preprocessor/core.py#L325
如果把这里的num_proc设置为None,就不会死锁在数据预处理阶段,但是会死锁在Train 0/100
如果关闭packing或者不使用多模态数据,就一切正常
Additional Information / 补充信息
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce with the provided swift sft command, focusing on --packing true with the multimodal dataset. Inspect swift/dataset/preprocessor/core.py around lines 325 and 353, then compare runs with packing disabled, multimodal data removed, or num_proc set to None. Done means preprocessing and training no longer deadlock under the reported configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100