modelscope / modelscope/ms-swift
Resolved: When fine-tuning DeepSeekOCR models using the ms-swift framework, training with mixed text and multimodal data cannot be processed.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the bug
在 swift/llm/template/template/deepseek.py 函数中，数据加载时，只将带有 images 和相应的 seq_mask 的数据取出进行处理；
如果同一个批次的数据（夹杂纯文本和多模态数据时），DeepseekOCRModel 的前向传播时，在模型处理图像特征后，通过：
inputs_embeds[idx].masked_scatter_(images_seq_mask[idx].unsqueeze(-1).to(device), images_in_this_batch)
将图像特征插入到带有占位符的文本特征中；
会导致无法准确地从批次数据里，找到对应的文本特征。
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in swift/llm/template/template/deepseek.py and inspect the data-loading function that selects examples with images and seq_mask. Trace how mixed text and multimodal batches reach the DeepSeekOCRModel forward pass and the masked_scatter_ operation. Done means a batch containing both kinds of data is processed without losing the correspondence between image features and text features.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100