mudler / mudler/vllm.cpp

port(SPEC-DRAFT-MODEL): async_scheduling_compatible refuses draft_model, which the pin advance makes wrong

Open
#2,649 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: SPEC-DRAFT-MODEL

Found by #2647 (PORT-NOW wave
PORTQ-3, tranche 81-120). Nothing was executed: this is a static reading of the
tree at e24ec8bfd against upstream c0202c5603 (vllm#48341).

The shape of this one is unusual, and it is the reason to file it

This tree is correct at the pin and becomes wrong the moment the pin advances.
It is the only entry in its tranche with that shape, and the tree already says so
in a comment.

include/vllm/config/speculative.h:514 is:

bool async_scheduling_compatible() const { return use_eagle(); }

and the comment directly above it, at :510-513, records the reconciliation by
name:

"draft_model" is refused AT THE PIN (b389ac2946 later allows it —
vllm/config/vllm.py:1291 there — but the pin governs; reconcile when the pin
advances past that commit).

c0202c5603 is the commit that does it. Its whole product delta is one conjunct
in vllm/config/vllm.py:

+                and self.speculative_config.method != "draft_model"

added to the branch that warns and disables async scheduling. c0202c5603 is an
ancestor of b389ac2946 (verified with git merge-base --is-ancestor), so the
comment's "that commit" resolves to this one or to an ancestor of it, and it is
inside the 5559679229..e126687a9a window the pin advance covers.

What to change

return use_eagle() || uses_draft_model();, plus the comment. uses_draft_model()
already exists at include/vllm/config/speculative.h:487 and "draft_model" is
an accepted --speculative-config method (:536). The predicate is reachable:
src/vllm/entrypoints/model_loader.cpp:2216-2217 consumes it to force async
scheduling off.

Size

~1 product line and its comment, plus a config-level test that a draft_model
speculative config no longer disables async scheduling. The test is the point:
without it the line is a silent default change.

Ordering

Do not land this before the pin advances past c0202c5603. Landing it early
makes the tree disagree with its own pin, which is the failure the comment at
:510-513 was written to prevent. It belongs in the pin-advance change, not
ahead of it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at include/vllm/config/speculative.h:487 and :510-514, then trace src/vllm/entrypoints/model_loader.cpp:2216-2217 to understand the predicate's effect. After the pin advances past c0202c5603, update the predicate and comment, and add a config-level test showing that draft_model no longer disables async scheduling. Do not land this before that pin advance.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.