ENG-MM-INPUT-PIPELINE: the five paths P2 made reachable without making complete
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: ENG-MM-INPUT-PIPELINE
P2 (#2379, landed by #2398) made the runner's multimodal path reachable: GPUModelRunner runs the registration's encode_mm per item and embed_mm per step and fills ModelForwardInput::mm, so a served Qwen3-VL image request is answered through ModelRegistry::Forward. It made five paths REACHABLE without making them COMPLETE, and each one refuses by name in the code rather than producing an answer.
Those five are listed under ## Owed in .agents/specs/multimodal-track.md (§"Owed by P2 — the runner multimodal path"). They pointed at #2379 as their owner, and #2398 CLOSES #2379 — so the owner would have been a closed issue. This issue is the live owner; #2379 stays as the history of how the path landed.
What is owed
- Qwen3-VL serves ONE request per step.
ForwardQwen3VLForConditionalGenerationreturns the last token's logits and does not readinput.logits_indices, so a batched step cannot be answered; it refuses withnum_reqs <= 1. Closing it is a per-row gather inside the VL forward (VLForwardLastLogitsDBuf), which moves the numbers the M2c golden was measured on and therefore needs its own gate run. - The merge pays a HOST round-trip.
EmbedMmQwen3VLForConditionalGenerationdownloads the gathered tower rows, runsQwen3VLMergeMultimodalandQwen3VLComputeDeepstackon the host in f32, and uploads the result. That is exactly the arithmetic the gated M2c driver runs, which is why it was chosen. A device-resident merge is a measured change against that golden, not a cleanup. - Only the
imagemodality reaches the runner.EncodeMmQwen3VL...refusesvideoandaudioby name. Qwen3-VL has a video tower and a video driver but no runner path: the video item's placeholder structure is timestamp-interleaved and needsQwen3VLGetRopeIndexVideo, a different M-RoPE entry point from the one the hook calls. - Gemma-4's
ForwardMmand Muse Glimmer's are still compile-only. Neither registration declaresencode_mm/embed_mm, soModelRegistry::SupportsMmInputsis false for them and the runner's multimodal arm is never entered. Declaring the hooks is the small half; the blocker is that neither model has a chat seam or a processor producingmm_features. - The seam install in
server_main.cpphas no reachability mutation of its own.chat.set_multimodal_chat_fn(...)is the production install and it landed before P2; the P2 e2e gate installs the same seam itself, because a unit test cannot driveserver_main's argv path. Deleting the production line leaves the P2 suite green.
Where the record lives
.agents/specs/multimodal-track.md ## Owed, block "Owed by P2 — the runner multimodal path". Each of the five entries names this issue as its owner.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ## Owed section in .agents/specs/multimodal-track.md, then inspect the named ForwardMm, EmbedMm, and EncodeMm entry points, ModelRegistry::SupportsMmInputs, and the seam in server_main.cpp. Run the existing P2 e2e gate and compare changes against the M2c golden. Done means the five listed paths work without their current refusals and the relevant gates still pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100