modelscope / modelscope/ms-swift
[Question/Feature Request] Support for training Qwen-Omni with full multimodal I/O (Text+Image+Audio -> Text+Audio)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Hello,
I am currently using ms-swift and I would like to inquire about the support status for the Qwen-Omni model.
Specifically, I aim to fine-tune the model with the following multimodal settings:
Input: Text + Image + Audio
Output: Text + Audio
I understand that ms-swift already supports Qwen2-VL (Text+Image) and Qwen2-Audio (Text+Audio), but I am not sure if the framework currently supports the end-to-end training of Qwen-Omni where the model generates audio directly from mixed inputs (including images).
Questions
Is this specific multimodal configuration (Text+Image+Audio -> Text+Audio) currently supported in ms-swift?
If yes, could you please provide a brief example of the training script arguments or a configuration file?
How should the dataset be formatted for this specific task? (e.g., how to structure the JSONL to include both image/audio paths in the history and expect an audio response).
Thank you for your time and for this amazing framework!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by reviewing the existing Qwen2-VL and Qwen2-Audio training paths and dataset-format documentation referenced in the issue; done would mean establishing whether the requested Qwen-Omni configuration is supported and documenting the required arguments and JSONL structure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100