mudler / mudler/vllm.cpp

The C ABI has no multimodal request path for any model, and it needs a row: two models serve images and audio through the server, neither is reachable from include/vllm.h

Open
#2,862 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

include/vllm.h has no multimodal request path, for any model. This is the C ABI's one
named open capability gap, it is tree-wide rather than any model row's, and it needs a row
of its own.

Measured

The header exposes three multimodal engine-CONFIGURATION fields — language_model_only
(:649), limit_mm_per_prompt (:669), mmproj_path (:683) — and zero multimodal
request fields
. No vllm_mm_* symbol exists.

The header says so itself, twice, in contract language:

  • :204-216 — "THIS ABI HAS NO MULTIMODAL CHAT REQUEST PATH YET ... the part is dropped,
    its text siblings still form the prompt"
  • :288-293 — "THIS ABI STILL HAS NO MULTIMODAL REQUEST PATH"

No model sets a precedent, Qwen3-VL included. Eight architectures declare
supports_multimodal = true; only two carry encode_mm/embed_mm and register a chat seam
— Qwen3-VL (qwen3_vl_registry.cpp:55, :505-506) and dots3-note
(dots3_note_registry.cpp:97). Both reach their tower only through
ApiServer::handle_chat_completions.
The seam interface itself
(REGISTER_VLLM_MM_CHAT, include/vllm/entrypoints/openai/mm_chat_registry.h:143-151) is an
in-tree C++ seam and is not shipped: CMakeLists.txt:2969 installs include/vllm.h alone,
and :2893-2894 states a consumer gets that header only.

Media DOES cross the ABI — but into separate handles, never into a language model:
transcription PCM (:1018-1021), video-generation PPM/WAV paths (:1160-1164), speech
reference audio (:1299-1301).

The docs are honest about it, which is why this is a gap rather than a defect:
docs/FEATURES.md:406 reads | Multimodal input (image/audio/video) | none | embedder-unreachable |, docs/guides/multimodal-input.md:83-85 says an image_url sent
through the C ABI is dropped, and tests/capi/test_capi.cpp:1457-1493 pins that behaviour.

Why this is not a model row's job

AGENTS.md §"Shared seams" requires every shipped capability to be reachable through
include/vllm.h, with examples and servers as thin clients. Two models now serve images and
audio through the server and neither is reachable from the ABI.

A first multimodal request path must serve Qwen3-VL as well as dots3-note, or it is a
per-model ABI — which is the thing the seam registry (#2481) was built to prevent. It also
owns scripts/abi-capability-allowlist.txt and scripts/check-surface-coverage.py, and it
changes docs/FEATURES.md. None of that belongs to a model port.

Two record repairs it should carry

  • scripts/abi-capability-allowlist.txt cites vllm.h:440,451 for
    vllm_chat/vllm_chat_stream; they are now include/vllm.h:988 and :999. Its reason
    text ("even the server's Qwen3-VL image seam does not run the vision tower in the live
    engine step") is falsified by tests/vllm/entrypoints/openai/test_api_server_mm_forward.cpp.
    Its VERDICT (ABI-unreachable) remains correct; only the anchors and the reason are stale.
  • include/vllm.h:290 names vllm_generate in a permanent contract paragraph. No such
    symbol exists — the completion entry point is vllm_complete (:866).

Unrelated, found while reading, recorded so it is not lost

examples/tokenize/main.cpp:20-21 includes internal headers, violating AGENTS.md's rule
that examples are thin clients of include/vllm.h. Not this row's work, but nothing else
tracks it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the multimodal contract sections in include/vllm.h, the Qwen3-VL and dots3-note registry seams, and include/vllm/entrypoints/openai/mm_chat_registry.h. Read tests/vllm/entrypoints/openai/test_api_server_mm_forward.cpp and tests/capi/test_capi.cpp to understand current server behavior and ABI behavior. Done means both models have an ABI request path, the allowlist anchors and docs are updated, and the existing coverage reflects the new contract.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.