mudler / mudler/vllm.cpp

port(ENG-SCHED-CORE): no drafting-slot reservation, so a spec-decode step schedules more input tokens than vLLM does

Open
#2,685 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: ENG-SCHED-CORE

Found by wave PORTQ-4 (#2680)
re-deriving PORT-NOW entries 129 and 156 of 5559679229..e126687a9a — upstream
0914ed2e81 vllm#51725 and
2ac1f683f1 vllm#51256.

They are filed together because they are one unit of work. [156] adds the
property and [129] is its only consumer; porting [156] alone lands dead code, and
porting them apart installs a subtraction that [129] then removes.

What is wrong, and why the queue understates it

Under speculative decoding this tree schedules more input tokens per step than
vLLM does
, because it has no drafting-slot reservation at all.

  • include/vllm/config/scheduler.h:166-168ResolvedMaxNumScheduledTokens() is a
    bare max_num_scheduled_tokens.value_or(max_num_batched_tokens) with no drafting
    delta.
  • new_slots and input_budget are 0 hits across src/, include/ and
    tests/; the positive control token_budget returns 26 hits in src/ and
    include/, so the zeros are real.
  • Every site upstream's post-commit input_budget touches has a token_budget
    twin here, which gives the port a clean mirror target:
    src/vllm/v1/core/sched/scheduler.cpp:640,653,687,748,791,855,907,910,973,1008.

This is a pre-pin hole, not a two-commit lag. Upstream already carried
max_num_new_slots_for_drafting and its subtraction at 5559679229:
git show 5559679229:vllm/config/speculative.py:1292-1305 for the property and
5559679229:vllm/config/vllm.py:1694-1743,1704,1734 for its two consumers. A queue
derived from a commit range structurally cannot see this shape.

The divergence is already recorded in a committed golden

tests/parity/goldens/dflash_27b/d0_blocked_traceback.txt:28 captures the pinned
oracle logging, on this project's own DFlash 27B gate configuration:

max_num_scheduled_tokens is set to 4036 based on the speculative decoding settings.

while this tree schedules the full max_num_batched_tokens. This is a reading of
a committed file, not a measurement.
Nothing was run. But it means any DFlash
throughput comparison against that oracle has been comparing unequal per-step token
budgets, and whoever ports this should re-check the affected benchmark rows.

What to build

  1. SpeculativeConfig::MaxNumNewSlotsForDrafting() in
    include/vllm/config/speculative.h, implementing the post-2ac1f683f1
    table: dflash to K, parallel + draft_model to K, parallel to K-1, draft_model to
    1, else 0. It belongs beside NumLookaheadTokens() at :520-529, which already
    encodes the DFlash k + 1 versus DSpark k asymmetry. Every input exists:
    speculative.h:100 parallel_drafting (set :158 dflash, :346 dspark),
    :467, :479-482, :487, :491, :498. ~20 lines.
  2. A per-step input_budget in Scheduler::Schedule, charged
    max_num_new_slots_for_drafting per scheduled request. ~15-20 lines plus one
    stored member near include/vllm/v1/core/sched/scheduler.h:328.
  3. The post-commit refusal when max_num_batched_tokens <= scheduled_token_delta.
    ~10 lines.
  4. The ported upstream table test — tests/test_config.py adds eight cases
    (eagle3/0, p-eagle/7, dflash/8, dspark/7, mtp/0, ngram/0, draft-model/1, pard/8).

Total roughly 60-80 lines of product code plus tests.

Owed

The PARD arm (parallel_drafting and uses_draft_model together) may not be
constructible here — src/vllm/v1/spec_decode/draft_model_proposer.cpp never sets
parallel_drafting — so that row of the table would land inert. It is part of the
ported artefact and should be named rather than dropped. SPEC-DRAFT-MODEL
(.agents/engine-matrix.md:194) owns that arm; SPEC-DFLASH (:179) and
SPEC-DSPARK (:181) own the polarity the table encodes.

What was not determined

Whether the tree ever overflows today. Upstream's reservation protects the
worker's token capacity when the drafter inserts mask queries; whether this tree's
DFlash runner has an independent cap that makes the overflow unreachable was not
settled statically and needs a run.

Nothing in this wave was executed — no build, no test run, no GPU, no lease.
Every path:line was printed from the tree at a700e8da6; the scheduler and
speculative-config anchors were re-read by the wave operator.

A correction the record owes

.agents/sync/2026-09-01-cdefd9d.md:1985-1990 reaches the same conclusion and is
corroborated here, but cites include/vllm/config/scheduler.h:165-167. The
resolver's body is at :166-168. Off by one, substance correct.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with SpeculativeConfig::MaxNumNewSlotsForDrafting() beside NumLookaheadTokens() in include/vllm/config/speculative.h, then trace Scheduler::Schedule and the scheduler state near include/vllm/v1/core/sched/scheduler.h:328. Compare the listed scheduler.cpp sites with the upstream behavior, add the eight cases to tests/test_config.py, and run the relevant tests; done means drafting reservations and refusal behavior are covered without dropping the inert PARD row.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.