mudler / mudler/vllm.cpp

gfx1151 decode: the 71.3% board-idle figure is a whole-leg average; windowed to the generate interval the board is 100% busy at 2872 MHz

Open
#3,015 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-GATE-ROCM-LLAMACPP

PR #3002's evidence records the board at 71.3% busy pooled (63.2% on the
64-token legs, 76.3% on the 128-token legs). That figure was read as the
strongest lead in the run: a compute-bound decode should not be idle a third of
the time, which points at host-side stalls or serialization.

It is a windowing artifact. The 71.3% is a mean over the whole leg, and a
leg contains process start, model load and teardown as well as the four timed
generations. job.log records generate_start_unix and generate_end_unix for
every run to the microsecond, and the clock sampler runs at --interval 0.25,
so the samples can be windowed to the generation intervals exactly.

Windowed to the generate intervals, with the cold run 1 dropped as the fold
drops it, over all 8 legs of /mnt/nas_share/rc/strix-arm-2933/out/:

leg samples busy mean sclk mean
n128-r1..r4 285 / 281 / 279 / 284 100.0 each 2877 / 2872 / 2870 / 2866
n64-r1..r4 143 / 144 / 144 / 143 100.0 each 2889 / 2875 / 2870 / 2871
pooled 1703 100.0 ~2872 MHz

gpu_busy_percent is integer-valued and 1703 of 1703 samples read exactly
100
. Minimum, p10, median and maximum are all 100.

The same treatment applied to llama.cpp

The survey's llama.cpp legs (/mnt/nas_share/rc/strix-survey-2497/out/survey-20260904/clock-llamacpp-r*.jsonl)
have no per-run timestamps, but the generation phase is unambiguous in the
series -- a single contiguous run of >= 95 samples in every leg. Over that
span:

leg samples span busy mean sclk mean
r1 52 12.8 s 99.3 2675
r2 54 13.3 s 99.3 2637
r3 54 13.3 s 99.3 2675
r4 54 13.3 s 99.2 2656

We hold the board at a higher occupancy and a 210 MHz higher shader clock than
llama.cpp does, and we are still 2.26x slower.
That is the opposite of the
signature a host-side stall leaves. It is the signature of a shader-bound
kernel: llama.cpp's lower sclk is what a DRAM-bound decode looks like on an
APU whose SMU drops the shader clock while the memory controller is the
critical path, and our pinned 2872 MHz is what a decode looks like when the
shaders are the critical path.

What this rules out, and what it does not

The counter is a firmware activity average sampled every 0.25 s. All samples
reading the integer 100 bounds serialized host-side idle at under about 0.5%
of decode wall time
, so per-token host work -- the hipStreamSynchronize in
greedy_argmax_host (src/vllm/v1/sample/sampler.cpp:275), host
detokenization (src/vllm/v1/engine/detokenizer.cpp:448-456), the depth-1
engine loop from max_concurrent_batches=1 -- cannot be more than a percent or
so of the gap at this model size. It does not resolve microsecond gaps
between the ~700 eager kernel launches per token, which no ROCm build can
collapse today (src/vllm/platforms/rocm.cpp:91-97 leaves
support_static_graph_mode() false). This tree's own same-binary A/B on
gfx1200 measured graph capture at 0-2%, sys time unchanged
(.agents/specs/rocm-decode-graph.md D7), which is consistent.

So at least 98% of the 103.5 ms/token excess is GPU kernel time.

What is asked

  1. Window the clock record to the generate interval in the arm harness's
    fold, or publish both numbers with the whole-leg one labelled as including
    load. The current figure has already misdirected one investigation.
  2. Run rocprofv3 --kernel-trace on gfx1151 for this artifact, both
    engines, one lease, identical workload, as .agents/porting.md requires.
    No per-kernel attribution exists on this board; #1910's table is gfx1200 on
    a 9B model. Without it, the split of the ~102 ms/token among the kernel
    families named in the linked issues stays a byte-weighted estimate.

Correctness state carried, unchanged, per #2921: TOKEN_GATE=FAIL, 3 of 6
prompts divergent against llama.cpp b10451, 5 of 6 against vLLM compiled,
every divergence a near-tie at about 0.125 nats, no deterministic denominator
established on this path. Nothing here is a parity claim.

Related: #2497, #2921, #2940, #3002, #2109, #1910, #2964, #332.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the arm harness fold that reads generate_start_unix and generate_end_unix from job.log, and review .agents/porting.md for the required profiling command. Window clock samples to generation intervals, preserve the whole-leg value with an explicit label, then run rocprofv3 --kernel-trace for both engines on gfx1151 with the identical workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.