mudler / mudler/vllm.cpp

Decode is 96% GPU-busy and 96.4% of that is one GEMV, so the gap is kernel time not scheduling

Open
#2,964 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: SPEC-DFLASH2

G4 ran, and it refutes the async premise

.agents/specs/dflash2-async-spec-sampler.md defines G4 and states its own verdict: "GPU busy at or above 95% of wall refutes the throughput half outright."

Measured 2026-09-05 on dgx:gpu0, tree ecf580ce9, evidence /mnt/nas_share/rc/dflash2-staged/out-g4/:

node mode graph mode
DECODE GPU busy % of wall 96.01 96.53
idle 190 ms / 4.765 s 156 ms / 4.497 s
steps 33 17
idle per step 5.77 ms of 144.4 ms 9.19 ms of 264.5 ms

There is about 4% idle on the device. The A2 chain removes host-side serialization, so it cannot recover the 11.1% we are behind (the gap is step cost). The per-step gap is a single consistent ~9.2 ms; removing all of it buys roughly 3.5%.

What the trace says the gap IS

From the same run's --cuda-graph-trace=node export:

activity count per step GPU ms % of busy
Kernel2 6471 196.1 4410.1 96.40
GdnSpecDecodeFusedKernel 768 23.3 100.1 2.19
ConvSpecUpdateKernel 768 23.3 10.4 0.23
PagedFlashKernel 90 2.7 2.7 0.06
flash_fwd_splitkv_kernel 272 8.2 2.3 0.05

Top launch geometries, all grid [8, N, 1] with block [32, 1, 1] — one warp per block:

geometry n per step GPU ms % of busy
[8, 272, 1 / 32,1,1] 1178 35.7 1850.3 40.45
[8, 40, 1 / 32,1,1] 2373 71.9 1327.1 29.01
[8, 128, 1 / 32,1,1] 816 24.7 604.1 13.20
[8, 1940, 1 / 32,1,1] 18 0.55 204.9 4.48

Attention is not the cost. One GEMV is. We are device-saturated and GEMV-bound, and vLLM runs 263 ms/step against our 292 ms on the same weights, prompts and box.

What is asked

  1. Profile the vLLM arm. The G4 run traced it but produced no export — RESULTS.txt ends with vLLM graph-mode: last_leg_secs= tok_s=, empty. The .nsys-rep files exist (vllm-graph.nsys-rep, vllm-node.nsys-rep); the JSON does not. Without their kernel breakdown there is no cross-arm comparison, and .agents/porting.md requires both sides traced with the SAME tool on an identical workload before any invocation-parity claim.
  2. Identify what Kernel2 is and which upstream kernel vLLM resolves to for the same GEMM shapes, with the dtype, compute type, scale type, entry point, algorithm policy and resolved template dtypes that .agents/porting.md demands.
  3. Test the occupancy hypothesis. A 32-thread block is one warp; on GB10 that is a concrete under-occupancy candidate for a bandwidth-bound GEMV. It is a hypothesis, not a finding — the trace shows the geometry, not the reason.

Scope note

This supersedes the async chain as the row's throughput lever. A2-4 and A2-5 target idle that measurement says is not there; A2-5 is separately blocked on #2920. A2-1 through A2-3 stay landed for their structural value (they discharge each other's ## Owed and remove a reason-A hazard), and none of them claims a throughput result.

Measurement discipline for whoever takes this

Run variance here is real: vLLM read 16.111 then 16.323 across two runs, SGLang 16.034 then 15.683. A delta under about 2% at c=1 is noise. --gpu-metrics-devices is refused on this box (ERR_NVGPUCTRPERM); the kernel trace still exports, so read CUPTI_ACTIVITY_KIND_KERNEL and do not wait on GPU metrics sampling.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .agents/specs/dflash2-async-spec-sampler.md and .agents/porting.md, then inspect the vLLM .nsys-rep files and the missing JSON referenced by RESULTS.txt. Identify Kernel2 and the matching upstream kernels, recording the requested types, entry points, algorithm policy, and template dtypes. Test whether the one-warp geometry affects occupancy, using identical workloads and the stated variance threshold; done means a comparable kernel breakdown and supported occupancy conclusion.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.