mudler / mudler/vllm.cpp

SPEC-DFLASH2 A2-5: lift the async input-combine veto, the wave that captures the idle

Open
#3,023 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: SPEC-DFLASH2

Spec: .agents/specs/dflash2-async-spec-sampler.md, wave A2-5 — "flip the veto, and only here. The flip is one line at two sites; every wave above exists so that line is safe."

Why it is justified now, and why the arithmetic changed

The row's target was mis-stated and is now corrected. The evidence gives ours 14.914 vs vLLM 16.323 tok/s — a 9.45% throughput gap, not the 11.1% per-step-time ratio quoted earlier. Those differ because we accept MORE per step (4.358 vs 4.295 mean acceptance length), so a slightly longer step still wins on tokens/second.

throughput gap to close 9.45%
step-time reduction that achieves it at our acceptance 8.63%
decomposition ceiling (idle 3.99% + non-GEMM 3.46% + GEMM 2.61%) 10.06%
perfect levers 16.581 tok/s vs 16.323 — ahead by 1.58%

The known levers suffice. Idle is the largest single one at 3.99%, and A2-5 is the wave that captures it. Recorded in .agents/benchmark-record.md at 987b74ca6, with the five figures that did not reproduce from the evidence named there.

Every blocker is landed

  • A2-1 (e64f00560) — the draft-aware combine and its G2 instrument.
  • A2-2 (7be09d9f3) — device-resident verify; the accept walk's D2H moved to a copy queue.
  • A2-3 (03736b2fe) — device-resident propose; the draft buffer the combine reads.
  • #2920 (18157e04a) — the async decode arm proposes, so a spec engine's first step no longer drafts nothing.
  • A2-4 (4269469da) — the propose reads committed ids where they live. The committed_ids_on_host refusal is DELETED, not bypassed: DownloadCommittedIds runs unconditionally outside every write-back branch. This is what made the veto liftable.

What is asked

Lift the veto at both GPUModelRunner construction sites, and only that. Every wave above exists so this line is safe.

The gates this wave owes, from the spec

  • G3 (GPU). TPOT and output-throughput A/B at c=1 and c=8 on the #1574 DFlash2 workload, operator-run under an rc lease, against the same binary with VT_ASYNC_RUNNER=0. Spread at c=8 is about 5.9% and at c=1 about 1.6%, so a delta under the floor is reported as under the floor and never as a ratio.
  • G4 is RUN: 96.01% busy node mode, 96.53% graph mode, recorded at 987b74ca6.

Hazards, all of which this row has paid for

  • Reason A has been armed twice and disarmed twice. A2-3 re-armed it by passing cu_num_logits = nullptr from both CUDA arms, which drives logits_start to query_end - 1 — the last draft slot on a verify step. Lifting the veto is exactly what makes such a defect live, and it is invisible to every token gate because verify is lossless and only acceptance moves. Check both CUDA arms pass the real cu_num_logits before flipping anything.
  • Predicate split. A per-request refusal paired with a per-step route predicate shipped here as a silent wrong answer and survived 27 mutations because every test used num_reqs == 1 (#2710).
  • Gates that measure nothing. Three separate A2-3 reviews each found a guarantee no test could red.
  • The emit-count discriminator is not a committed gate, because reaching the pre-repair shape needed the veto deleted. Once the veto is lifted, that discriminator becomes constructible — and it is the instrument that separates a correct combine from reason A.

What this wave may not claim

No throughput result until G3 is measured. The 10.06% is a perfect-case ceiling against our own wall, not a measured gain, and RECORDED, no floor declared is not a pass.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read .agents/specs/dflash2-async-spec-sampler.md and inspect both GPUModelRunner construction sites, then verify both CUDA arms pass the real cu_num_logits before changing the veto. Done means the veto is lifted only at those two sites and G3 A/B measurements at c=1 and c=8 are run against VT_ASYNC_RUNNER=0; do not claim a throughput result before measuring.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.