modelscope / modelscope/ms-swift

[Question/Feature Request] Support for training Qwen-Omni with full multimodal I/O (Text+Image+Audio -> Text+Audio)

Open
#6,726 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Hello,

I am currently using ms-swift and I would like to inquire about the support status for the Qwen-Omni model.

Specifically, I aim to fine-tune the model with the following multimodal settings:

Input: Text + Image + Audio

Output: Text + Audio

I understand that ms-swift already supports Qwen2-VL (Text+Image) and Qwen2-Audio (Text+Audio), but I am not sure if the framework currently supports the end-to-end training of Qwen-Omni where the model generates audio directly from mixed inputs (including images).

Questions

Is this specific multimodal configuration (Text+Image+Audio -> Text+Audio) currently supported in ms-swift?

If yes, could you please provide a brief example of the training script arguments or a configuration file?

How should the dataset be formatted for this specific task? (e.g., how to structure the JSONL to include both image/audio paths in the history and expect an audio response).

Thank you for your time and for this amazing framework!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by reviewing the existing Qwen2-VL and Qwen2-Audio training paths and dataset-format documentation referenced in the issue; done would mean establishing whether the requested Qwen-Omni configuration is supported and documenting the required arguments and JSONL structure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.