fix(SCHED-ASYNC-PREEMPT): a preemption leaves stale placeholders and the count goes negative under NDEBUG
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: -
Owed by .agents/specs/upstream-sync-pinport.md
## Owed. Found while re-deriving the PORT-NOW queue for the cdefd9d499 sync
cycle (#2524); listed in the sync report §13 as one of two live defects.
The bug
Under async scheduling, any preemption can drive num_output_placeholders
negative.
Scheduler::preempt_request resets the request but never clears its in-flight
placeholder count:
src/vllm/v1/core/sched/scheduler.cpp:345-382 (at 63889449c) sets
status = RequestStatus::kPreempted, increments num_preemptions, and resets
num_computed_tokens to 0. It does not touch num_output_placeholders.
AsyncScheduler::update_request_with_output then drains unconditionally:
src/vllm/v1/core/sched/async_scheduler.cpp:93-95
// async_scheduler.py:66-68: drain the placeholder count by the accepted tokens.
request.num_output_placeholders -= static_cast<int>(kept.size());
assert(request.num_output_placeholders >= 0);
A request preempted while output was in flight keeps a stale placeholder count.
When that output lands, the decrement takes the count below zero.
The assert does not save a release build. It is a bare assert, so NDEBUG
compiles it out and the count simply goes negative in silence. The negative value
then feeds the async max-tokens guard at
src/vllm/v1/core/sched/scheduler.cpp:663, which reads
num_computed_tokens + 2 - num_output_placeholders, so a negative placeholder
count makes that expression LARGER and can retire a request early.
Upstream
Fixed by vllm#48245 at
a0c092ee72, an ancestor of the sync target cdefd9d499. Upstream replaced the
whole mechanism: num_stale_output_tokens / drop_stale_output, zeroing the
placeholders on every preemption, and skipping the decrement when the output
is stale.
Our tree still carries the superseded async_tokens_to_discard one-per-call
drain (async_scheduler.cpp:80-81, guard read at scheduler.cpp:1215). Nothing
in src/ ever sets it above zero; only
tests/vllm/v1/test_async_scheduler.cpp:655 does.
Notes for whoever ports it
- We have no
num_in_flight_tokensfield onRequest
(include/vllm/v1/request.h:263-271). - Upstream's "block resume while deliverable stale output is in flight" arm rides
onskipped_waiting, whichinclude/vllm/v1/core/sched/scheduler.h:67-68
records as deferred here. That arm may not be portable yet; say so rather than
approximating it. - The reproduction needs async scheduling plus a preemption, so the smallest
failing test belongs besidetests/vllm/v1/test_async_scheduler.cpp. Assert on
the placeholder count itself, not on theassertfiring: a release build will
not trap.
Not fixed in the finding flow: this wave could not compile. The dev box was at
load ~50 with two other worktrees building and ~1 GiB free, and a third
concurrent build is this repository's recorded OOM failure.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing src/vllm/v1/core/sched/scheduler.cpp and async_scheduler.cpp with upstream vllm#48245, then read Request in include/vllm/v1/request.h and the scheduler state in scheduler.h. Add a regression beside tests/vllm/v1/test_async_scheduler.cpp that exercises async preemption and checks the placeholder count in a release-style build. Done means stale output cannot make the count negative and supported upstream behavior is covered without assuming unavailable fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100