modelscope / modelscope/ms-swift
MiniCPM-V 4.6 training hangs on text-only samples with DeepSpeed
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。
Bug Description / Bug 描述
当使用 DeepSpeed ZeRO 训练 MiniCPM-V 4.6 时，如果数据集中包含纯文本样本（没有图片/视频），训练过程会卡住。原因如下：
- 对于纯文本样本，
MiniCPMV4_6Model.forward()会完全跳过视觉编码器（vision_tower+merger），因为pixel_values和pixel_values_videos均为None。 - 在 DeepSpeed ZeRO 下，参数被分片到多个 GPU 上，只有在计算触及这些参数时，才会通过 all-gather 按需收集。
- 当一个 GPU 处理纯文本样本（不涉及视觉计算）而另一个 GPU 处理图像样本（需要视觉参数）时，all-gather 同步会发生死锁——其中一方永远不会触发另一方正在等待的 gather 操作。
How to Reproduce / 如何复现
训练数据集中包含纯文本数据即可复现
Additional Information / 补充信息
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating MiniCPMV4_6Model.forward() and trace how vision_tower and merger are skipped when pixel_values and pixel_values_videos are None. Reproduce with DeepSpeed ZeRO using a dataset containing both text-only and image samples. Done means training no longer hangs when different GPUs process those sample types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100