mudler / mudler/vllm.cpp

perf(SAMPLE-BEAM): beam search detokenizes every intermediate step it discards

Open Beginner friendly
#2,684 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: SAMPLE-BEAM

Found by wave PORTQ-4 (#2680)
re-deriving PORT-NOW entry 127 of 5559679229..e126687a9a, upstream
8977ea8895 vllm#50333.

What is wrong

Beam search detokenizes every intermediate beam step it then throws away. The
per-beam step params are built once, at
src/vllm/entrypoints/beam_search.cpp:138-147, and never set detokenize:

  auto MakeStepParams = [&]() {
    SamplingParams sp;
    sp.logprobs = 2 * beam_width;
    sp.max_tokens = 1;
    sp.temperature = params.temperature;
    sp.PostInit();
    return sp;
  };

detokenize defaults to true (include/vllm/sampling_params.h:211) and is
honoured downstream, so setting it would actually skip the work:
src/vllm/v1/engine/output_processor.cpp:150-151 reads
const tok::Tokenizer* detok_tokenizer = sp.detokenize ? tokenizer : nullptr;.

The path is reached in production from both frontends —
src/vllm/entrypoints/openai/serving_completion.cpp:222-241 and
src/vllm/entrypoints/openai/serving_chat.cpp:720-742 — and both drivers delegate
to one BeamSearchDrive (beam_search.cpp:219-238), so there is exactly one
construction site to fix.

Size

One line, sp.detokenize = false;, plus a test asserting the per-beam request
carries it. Upstream's second hunk (offline.py:444, the allowed_token_ids beam
variant) has no counterpart here — this tree has a single step-params site — so that
half is surface-absent and collapses into the same line.

Verification state

Nothing was executed — no build, no test run, no GPU, no lease. Every
path:line was printed from the tree at a700e8da6 and re-read by the wave
operator. git show 5559679229:vllm/entrypoints/generate/beam_search/offline.py
lacks detokenize=False, so the commit adds it and this is not a pre-pin hole.
No prior triage exists: git log --all --grep 8977ea8895 is empty and the only
.agents/ reference is the queue line itself.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/vllm/entrypoints/beam_search.cpp at the MakeStepParams construction around lines 138-147, then read src/vllm/v1/engine/output_processor.cpp:150-151 to confirm how detokenization is selected. Locate the beam-search test coverage and add an assertion for the per-beam request. Done means the intermediate beam request disables detokenization and the relevant test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.