Tencent / Tencent/AngelSpec

[Feature] Support image-text multimodal input for training

Open
#1 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
270
Forks
28
PR merge metrics
No merged PRs in 30d

Description

Motivation

Training in AngelSpec is text-only today, so vision-language targets (Qwen3.5,
Qwen3-VL, Hy3-VL) can't be used. We'd like to train DFlash/DSpark drafters for
VLM targets on image-text multimodal input.

Some of the plumbing already exists (media extraction in the dataset, an
image_placeholder in templates, enable_multimodal in the inference config,
multimodal_inputs passed to vLLM). The gap is on the draft-training side,
where block-parallel drafters have no mRoPE support.

Proposed Solution

Support image-text records end to end: accept images in the training data, carry
the target's mRoPE positions through to the draft training forward, and make the
block-parallel draft RoPE mRoPE-aware.

The drafter core shouldn't need to change — KV injection, anchor sampling, and
the losses are all modality-agnostic — and since DFlash, DSpark, and DFly share
a backbone, one implementation should cover all three.

Happy to help implement or test.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing media extraction in the dataset, the image_placeholder in templates, and enable_multimodal in the inference config. Follow multimodal_inputs into vLLM, then inspect the draft-training forward and block-parallel draft RoPE path for carrying the target's mRoPE positions. Done means image-text records work end to end for the shared DFlash, DSpark, and DFly backbone.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.