mudler / mudler/vllm.cpp

defer(SPEC-ACCEPT-VARIANTS): the draft-logits cache dtype change is inert, and it makes a recorded layout stale

Open
#2,668 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: SPEC-ACCEPT-VARIANTS

Found by wave PORTQ-2 re-deriving PORT-NOW entry 80 of 5559679229..e126687a9a
(#2646). Upstream:
81bc196913 vllm#50910, "[Model Runner V2] Cache draft logits in model's LM head
dtype".

Deferred work with a named gate, filed so the person who lands the gate meets
it.

The gate: probabilistic draft sampling is refused by name

src/vllm/config/speculative.cpp:489-496 refuses any draft_sample_method other
than "greedy", and its message names this row:

"This engine drafts greedy only ... Probabilistic draft sampling is owed by row
SPEC-ACCEPT-VARIANTS"

The whole commit is about the draft_logits CACHE and the temperature algebra
around it. That cache exists upstream only when the draft samples
probabilistically, and it is consumed only by the stochastic rejection sampler.
Neither is reachable here: _cache_draft_logits is listed as step 6 and marked
"(probabilistic only; owed)" at
include/vllm/v1/worker/gpu/spec_decode/dflash2/speculator.h:13, and the
stochastic verify is deferred at
include/vllm/v1/spec_decode/rejection_sampler.h:53-57.

CORRECTION: this commit does NOT make our recorded layout stale

An earlier version of this issue said it did. That was wrong, and acting on it
would have corrupted a correct record.
It claimed 81bc196913 changes the
draft-logits cache to "the LM-head dtype holding pre-temperature logits", making
.agents/specs/dflash2-spec-decode.md's recorded (torch.float32, -inf) stale
on the pin advance. Verified at the advance target e126687a9a:

  • the commit's dtype hunk edits DraftModelSpeculator in the BASE
    vllm/v1/worker/gpu/spec_decode/speculator.py, whose draft_logits_spec
    returns model_config.head_dtype, 0.0 at :332-337;
  • DFlash2 has its own draft_logits_spec, and it still returns
    torch.float32, -float("inf")
    at
    vllm/v1/worker/gpu/spec_decode/dflash2/speculator.py:133-137, with the reason
    in a comment beside it;
  • DFlash2 writes that cache with its own _cache_draft_logits_kernel
    (dflash2/speculator.py:79-105, launched at :167), never through
    gumbel_sample's logits_cache, which is the parameter this commit renames;
  • the commit's file list is dspark/speculator.py, gumbel.py,
    rejection_sampler_utils.py and the base speculator.pynot
    dflash2/speculator.py.

So neither half reaches DFlash2. Our record at
include/vllm/v1/worker/gpu/spec_decode/dflash2/speculator.h:31,35-36,
include/vt/ops.h:1692 and .agents/specs/dflash2-spec-decode.md:852,1627 is
correct at the advance target. Nothing is owed here, and a pin-advancing wave
should leave those lines alone.

The inert disposition is unaffected and holds on its own ground, stated
above: probabilistic draft sampling is refused by name, so the cache this commit
re-dtypes cannot exist in any configuration this tree builds. What changed is the
reasoning attached to it, not the label.

Found by a fresh review of #2675; the wave's report carries the same retraction
at .agents/sync/2026-09-03-portq2.md §9.1.

Nothing was executed for this finding: no build, no test, no GPU.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading src/vllm/config/speculative.cpp:489-496 and the DFlash2 speculator files cited in the issue, then compare the upstream commit's file list. No patch or test is requested; the work is complete when the inert disposition and the recorded DFlash2 layouts remain verified and unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.