mudler / mudler/vllm.cpp

port(SERVE-REQUEST-LENGTH-GUARD): four generation-input bounds are missing, and beam_width bypasses the one that exists

Open
#2,688 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: SERVE-REQUEST-LENGTH-GUARD

Found by wave PORTQ-4 (#2680)
re-deriving PORT-NOW entry 137 of 5559679229..e126687a9a, upstream
b64a2708b0 vllm#51447
"Bound generation inputs before expensive work".

That title is this row's own thesis, which is why the row owns it.

Four bounds are missing, and one of them is reachable today

beam_width is entirely unvalidated, and the existing n bound does not cover
it.
BeamSearchParams is a plain aggregate with no validation
(include/vllm/entrypoints/beam_search.h:61-73), and beam_width is assigned
straight from the request n at
src/vllm/entrypoints/openai/protocol.cpp:741 (bp.beam_width = n;) and :752
(bp.beam_width = n.value_or(1);). The beam branch at
src/vllm/entrypoints/openai/serving_completion.cpp:222-241 calls
to_beam_search_params directly and bypasses to_sampling_params, so
SamplingParams::Verify's n > kMaxNSequences check
(src/vllm/sampling_params.cpp:48-55, kMaxNSequences = 16384 at
include/vllm/sampling_params.h:60) never runs on this path. An unvalidated n
reaches BeamSearchDrive.

The other three:

  • No dedup of stop_token_ids or bad_words. PostInit
    (src/vllm/sampling_params.cpp:243-257) builds a deduplicating std::set for the
    derived all_stop_token_ids, but leaves the source vectors as given.
  • No stop-string count bound. ParseStop
    (src/vllm/entrypoints/openai/protocol.cpp:100-107) takes the array unbounded;
    VLLM_MAX_STOP_STRINGS, max_stop_strings and kMaxStopStrings are 0 hits
    across src/, include/ and tests/, against a positive control of
    VLLM_MAX_N_SEQUENCES which is found at sampling_params.cpp:55 and
    sampling_params.h:59.
  • No bad-word count cap. src/vllm/sampling_params.cpp:165-176 validates only
    that no bad word is the empty string.

What is not portable

Upstream's v1/worker/gpu/sample/bad_words.py half is surface-absent. Its
MAX_BAD_WORDS_TOTAL_TOKENS / MAX_NUM_BAD_WORDS exist to size two fixed-width
GPU staging buffers. This tree has none: bad words are a dynamic nested container,
std::map<int, std::vector<std::vector<int32_t>>> bad_words_token_ids at
include/vllm/v1/sample/metadata.h:84, filled per slot at
src/vllm/v1/worker/gpu/input_batch.cpp:356-360 and read by
include/vllm/v1/sample/ops/bad_words.h:22. There is nothing to size. The
validation remains portable and is where the bound belongs here.

Upstream's envs.py half also differs in mechanism: this tree expresses such bounds
as compile-time constants (include/vllm/sampling_params.h:60), so the port is
three constants beside kMaxNSequences, not an env table.

Size

Roughly 40-70 lines: two dedups, one beam_width verify, one stop-count check in
both from_jsons, one bad-word count check, three constants, and tests.

Co-owning rows for the sampling-side halves: SAMPLE-BEAM
(.agents/engine-matrix.md:147) for beam_width, SAMPLE-LOGIT-FILTERS (:145)
for the dedup and bad-word cap — its anchors already name sampling_params.cpp
validation and input_processor.cpp bad-words tokenization.

A decision this needs, not just a port

Mirroring VLLM_MAX_STOP_STRINGS = 4 is a new, tight, user-visible refusal. It
may refuse requests this server serves today. Policy says mirror vLLM; the point is
that the refusal should be a stated decision in whichever spec claims this, not a
silent consequence.

Verification state

Nothing was executed — no build, no test run, no GPU, no lease. Every
path:line was printed from the tree at a700e8da6; the beam_width bypass chain
was re-read by the wave operator. b64a2708b0 is not an ancestor of the pin, and
the n bound it refactors predates the pin and is already mirrored here — which is
the correct pre-pin reading. No prior triage: git log --all --grep b64a2708b0 is
empty.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with include/vllm/sampling_params.h and src/vllm/sampling_params.cpp, then trace request parsing in src/vllm/entrypoints/openai/protocol.cpp and the beam branch in src/vllm/entrypoints/openai/serving_completion.cpp. Review existing sampling tests and add coverage for beam_width, duplicate inputs, stop-string counts, and bad-word counts. Done means all four bounds are enforced and the stop-string limit is explicitly decided and documented in the relevant spec.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.