perf(SAMPLE-BEAM): beam search detokenizes every intermediate step it discards
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: SAMPLE-BEAM
Found by wave PORTQ-4 (#2680)
re-deriving PORT-NOW entry 127 of 5559679229..e126687a9a, upstream
8977ea8895 vllm#50333.
What is wrong
Beam search detokenizes every intermediate beam step it then throws away. The
per-beam step params are built once, at
src/vllm/entrypoints/beam_search.cpp:138-147, and never set detokenize:
auto MakeStepParams = [&]() {
SamplingParams sp;
sp.logprobs = 2 * beam_width;
sp.max_tokens = 1;
sp.temperature = params.temperature;
sp.PostInit();
return sp;
};
detokenize defaults to true (include/vllm/sampling_params.h:211) and is
honoured downstream, so setting it would actually skip the work:
src/vllm/v1/engine/output_processor.cpp:150-151 reads
const tok::Tokenizer* detok_tokenizer = sp.detokenize ? tokenizer : nullptr;.
The path is reached in production from both frontends —
src/vllm/entrypoints/openai/serving_completion.cpp:222-241 and
src/vllm/entrypoints/openai/serving_chat.cpp:720-742 — and both drivers delegate
to one BeamSearchDrive (beam_search.cpp:219-238), so there is exactly one
construction site to fix.
Size
One line, sp.detokenize = false;, plus a test asserting the per-beam request
carries it. Upstream's second hunk (offline.py:444, the allowed_token_ids beam
variant) has no counterpart here — this tree has a single step-params site — so that
half is surface-absent and collapses into the same line.
Verification state
Nothing was executed — no build, no test run, no GPU, no lease. Every
path:line was printed from the tree at a700e8da6 and re-read by the wave
operator. git show 5559679229:vllm/entrypoints/generate/beam_search/offline.py
lacks detokenize=False, so the commit adds it and this is not a pre-pin hole.
No prior triage exists: git log --all --grep 8977ea8895 is empty and the only
.agents/ reference is the queue line itself.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/vllm/entrypoints/beam_search.cpp at the MakeStepParams construction around lines 138-147, then read src/vllm/v1/engine/output_processor.cpp:150-151 to confirm how detokenization is selected. Locate the beam-search test coverage and add an assertion for the per-beam request. Done means the intermediate beam request disables detokenization and the relevant test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100