modelscope / modelscope/ms-swift
训练多模态大模型的时候如果通过传入bbox或者mask在attention计算的时候，让模型更加关注页面的某一区域细节信息
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。
Bug Description / Bug 描述
在训练多模态大模型的时候，往往会发现模型会对于图片的一些细节信息感知不足，而这些关键信息往往又会影响结果。因此在使用ms-swift(Megatron)训练多模态大模型的时候，现在如何可以通过参数或者机制让梯度回传loss的时候，模型更加关注某一区域的tokens。
How to Reproduce / 如何复现
或者希望未来能支持这样的功能。
Additional Information / 补充信息
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the ms-swift (Megatron) multimodal training flow from loss backpropagation into attention, focusing on how bbox or mask information could identify image-region tokens. Done means a defined parameter or mechanism lets training emphasize the selected region and documents how to configure and verify that behavior; no files or tests are named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100