mudler / mudler/vllm.cpp

Five surfaces refuse compressed media for the same missing demuxer, and no row owns the codec brick mm-serving.md already proposes

Open
#2,814 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

Five surfaces across this tree refuse compressed media for the same reason -- nothing here
vendors a demuxer -- and each carries its own named residual. This issue proposes one shared
codec brick to discharge them, and records the evidence so the next reader does not
re-derive it.

The five

Surface What it says today
src/vllm/entrypoints/openai/server_main.cpp:1538-1540 image: "container-format decode (PNG/JPEG -> RGB) is a named MM-SERVE residual; supply raw RGB"
src/vllm/model_executor/models/ltx2_audio_input.cpp:43-45 "this project vendors no demuxer, so MP3, FLAC, OGG and MP4 audio are refused rather than misread"
.agents/specs/ltx25-a2v-audio-input.md:465 "Compressed audio containers -- refused by name; no demuxer is vendored."
src/vllm/entrypoints/openai/speech_api.cpp:193-196 the ENCODE side: "no mp3/opus/aac/flac encoder is vendored"
.agents/specs/dots3-note.md ## Owed dots3-note W7c's container arm

.agents/specs/mm-serving.md already proposes the right shape for this in prose: "a small
codec brick."
Nothing has owned it.

Why it is not a model row's job

The surface is unbounded. libsndfile 1.2.2 reports 26 container formats, and upstream's
own fallback is PyAV/ffmpeg, which adds hundreds of demuxers
(vllm/multimodal/media/audio.py:216-258 @ 9035151d6: soundfile first, PyAV on
ImportError or a LibsndfileError whose code is in _BAD_SF_CODES).

A verified inventory of what this tree has: zero audio dependencies of any kind. No
libsndfile, no libav*/ffmpeg linkage, no libswresample, no dr_wav, stb_vorbis, minimp3,
libFLAC, opus, ogg, soxr, libsamplerate, speexdsp. third_party/ holds six entries -- blake3,
doctest, httplib, minja, nlohmann/json, Vulkan headers -- none of them audio. ffmpeg appears
only as an external process whose argv the library composes and never spawns
(src/vllm/model_executor/models/minimax_h3_mux.cpp:12).

The decision this needs

It turns on the archive contract's no-vendor rule, which is project policy rather than a
row decision. Note the precedent already exists for a deliberate exception:
docker/Dockerfile:209-215 installs the ffmpeg BINARY as a documented divergence from that
rule.

Options, roughly in ascending cost: vendor a narrow decoder set (dr_wav / minimp3 / stb_vorbis
are single-header and permissively licensed, but that is three formats, not 26); link
libsndfile optionally and refuse by name when absent; keep refusing permanently and document
raw PCM/RGB as the contract.

One record repair that should ride along

.agents/specs/ltx25-retake.md's ## Owed points at issue #924 for "A container demuxer.
Without it the audio half of retake cannot exist"
-- and #924 does not resolve. That
tracking number is dangling and needs reconciling whichever way this issue is decided.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .agents/specs/mm-serving.md and compare the five residuals in server_main.cpp, ltx2_audio_input.cpp, ltx25-a2v-audio-input.md, speech_api.cpp, and dots3-note.md. Review the cited audio fallback and Dockerfile precedent before choosing a policy option. Done means the codec brick has an owner and the dangling reference to issue #924 is reconciled.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
audio-video-rtc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.