modelscope / modelscope/ms-swift

支持任意 ViT Encoder 与任意 LLM 自由组合构建多模态模型

Open
#9,813 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Checklist / 检查清单
  • I have searched existing issues, and this is a new feature request. / 我已经搜索过现有的 issues,确认这是一个新的 Feature Request。
Feature Request Description / Feature Request 描述

冒昧提一个需求建议:不知 ms-swift 能否参考 LoongForge 的设计,支持通过配置文件或命令行参数,将代码库中已支持的任意 ViT 编码器(如 CLIP-ViT、Qwen-ViT、InternViT 等)与任意 LLM(如 Qwen、LLaMA、InternLM 等)自由组合成多模态模型并直接训练,同时仍能对 ViT、Aligner、LLM 各自独立控制冻存与学习率?目前虽有 MultiModelKeys 注册机制,但每新增组合仍需手写适配代码,若能做到配置驱动的即插即用,对我们做实验对比会非常方便。当然,这只是个人的一点想法,不一定符合项目的设计方向,如果团队有别的考量也完全理解,还是非常感谢 ms-swift 已经提供的强大支持!

Pull Request / Pull Request 信息

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing MultiModelKeys registration mechanism and the current model-combination adapters mentioned in the issue. Define how configuration or CLI parameters would compose supported ViT encoders with LLMs and independently control ViT, aligner, and LLM freezing and learning rates; done means arbitrary supported combinations can train without handwritten adapters.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.