mudler / mudler/vllm.cpp

fix(SCHED-ASYNC-PREEMPT): a preemption leaves stale placeholders and the count goes negative under NDEBUG

Open
#2,531 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

Owed by .agents/specs/upstream-sync-pinport.md
## Owed. Found while re-deriving the PORT-NOW queue for the cdefd9d499 sync
cycle (#2524); listed in the sync report §13 as one of two live defects.

The bug

Under async scheduling, any preemption can drive num_output_placeholders
negative.

Scheduler::preempt_request resets the request but never clears its in-flight
placeholder count:

src/vllm/v1/core/sched/scheduler.cpp:345-382 (at 63889449c) sets
status = RequestStatus::kPreempted, increments num_preemptions, and resets
num_computed_tokens to 0. It does not touch num_output_placeholders.

AsyncScheduler::update_request_with_output then drains unconditionally:

src/vllm/v1/core/sched/async_scheduler.cpp:93-95

// async_scheduler.py:66-68: drain the placeholder count by the accepted tokens.
request.num_output_placeholders -= static_cast<int>(kept.size());
assert(request.num_output_placeholders >= 0);

A request preempted while output was in flight keeps a stale placeholder count.
When that output lands, the decrement takes the count below zero.

The assert does not save a release build. It is a bare assert, so NDEBUG
compiles it out and the count simply goes negative in silence. The negative value
then feeds the async max-tokens guard at
src/vllm/v1/core/sched/scheduler.cpp:663, which reads
num_computed_tokens + 2 - num_output_placeholders, so a negative placeholder
count makes that expression LARGER and can retire a request early.

Upstream

Fixed by vllm#48245 at
a0c092ee72, an ancestor of the sync target cdefd9d499. Upstream replaced the
whole mechanism: num_stale_output_tokens / drop_stale_output, zeroing the
placeholders on every preemption, and skipping the decrement when the output
is stale.

Our tree still carries the superseded async_tokens_to_discard one-per-call
drain (async_scheduler.cpp:80-81, guard read at scheduler.cpp:1215). Nothing
in src/ ever sets it above zero; only
tests/vllm/v1/test_async_scheduler.cpp:655 does.

Notes for whoever ports it

  • We have no num_in_flight_tokens field on Request
    (include/vllm/v1/request.h:263-271).
  • Upstream's "block resume while deliverable stale output is in flight" arm rides
    on skipped_waiting, which include/vllm/v1/core/sched/scheduler.h:67-68
    records as deferred here. That arm may not be portable yet; say so rather than
    approximating it.
  • The reproduction needs async scheduling plus a preemption, so the smallest
    failing test belongs beside tests/vllm/v1/test_async_scheduler.cpp. Assert on
    the placeholder count itself, not on the assert firing: a release build will
    not trap.

Not fixed in the finding flow: this wave could not compile. The dev box was at
load ~50 with two other worktrees building and ~1 GiB free, and a third
concurrent build is this repository's recorded OOM failure.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing src/vllm/v1/core/sched/scheduler.cpp and async_scheduler.cpp with upstream vllm#48245, then read Request in include/vllm/v1/request.h and the scheduler state in scheduler.h. Add a regression beside tests/vllm/v1/test_async_scheduler.cpp that exercises async preemption and checks the placeholder count in a release-style build. Done means stale output cannot make the count negative and supported upstream behavior is covered without assuming unavailable fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.