gfx1151: the vllm.cpp survey arm has never been measured on a post-#2511 head, so its fault rate is unknown
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-GATE-ROCM-LLAMACPP
The gfx1151 multi-engine survey published for #2921 reports the vllm.cpp arm as
FAULTED, 0 of 4, rc 139, and #2933 established why: the run measured
11fed3ba5, a revision that predates the fix for #2511. survey.sh asserted the
llama.cpp source manifest and refused to run when it moved, but for our own arm
it only checked [ -x "$VC" ] and printed the revision. It never built,
never fetched and never asserted, so it inherited #2511's staging directory at
/tmp/rocm-strix-q4k and measured that campaign's pre-fix build.
So the published fault rate is #2511 reproducing, not a regression.
What is still unmeasured
Whether the current head completes this workload on gfx1151 is not
established by that run and is not established by #2511 either. #2511's
"0 faults in 21 legs" was a different workload (--max-tokens 288 --repeat 1
and --max-tokens 64 --repeat 4 against a branch build) in a different lease.
Neither answers the survey's own question.
What this owns
One lease on strix:gpu0 that measures the vllm.cpp arm of the #2921 survey on
a binary built from main's head inside that lease, with the revision
asserted rather than printed, and reports the fault rate honestly whatever
it is.
Acceptance
- The job clones the declared revision, asserts
rev-parse HEADand
rev-parse HEAD^{tree}against values the caller declares, asserts the
worktree is unmodified, and exits non-zero when any of them disagrees.
Fail-closed is the whole point: #2933 exists because the check was aprintf. - The build directory is removed before configure, so a stale build cannot be
inherited the way/tmp/rocm-strix-q4kwas. The builtlibvllm.soand
vllm-clisha256 are recorded, and the run refuses the exact pre-fix
vllm-clisha256a703b83dd8954ba6dd3cbe82efcd38083c1d55492bbbaecf5c406f7c6efd646f
that #2933 names. - The workload is the survey's own:
Qwen3.8-27B-Q4_K_M.gguf, sha256
7e78da5d…fe169, 17,106,775,008 B verified on the worker,
--prompt 'The capital of France is' --max-tokens 64 --temperature 0 --repeat 4 --max-num-seqs 1, four interleaved rounds, run 1 of each leg cold
and discarded, an AMD clock window sampled beside every leg, and N taken from
the design rather than from a log tally. - A second token count (128) is timed in the same lease, so a decode-only figure
can be derived the same way the survey derived vLLM's, and labelled
derived wherever it appears. Without it our figure is whole-completion and
cannot be divided by llama.cpp'sllama-bench -p 0decode number: PR #2940
records that exact mismatch emitting a meaningless 1.814x. - The fault rate is published whatever it is. If the current head still faults,
that supersedes #2933's diagnosis and is reported as the result. - The correctness caveat travels with the number, on the face of the table:
TOKEN_GATE=FAIL, every divergence a near-tie at about 0.125 nats, no
deterministic denominator on this path. Those divergence counts are constants
carried from earlier evidence and are not re-measured here.
Not in scope
Re-measuring llama.cpp or vLLM. Their figures stand at 12.233 tok/s
(docs/bench-evidence/rocm-strix-llamacpp-denominator-20260902.md), 12.219 from
the survey's own legs, and 6.734 tok/s whole completion / 11.056 derived decode.
Repairing survey.sh itself, which is #2933's own acceptance.
Related: #2933, #2497, #2921, #2511, #2940.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the BACKEND-GATE-ROCM-LLAMACPP survey row and the existing vllm.cpp job entry point, including survey.sh for its current revision checks. Run the workload on strix:gpu0 with a clean build directory, verify the declared Git revision, tree, worktree, binaries, model, clock window, and token counts, then publish the fault rate and derived decode figure with the TOKEN_GATE=FAIL caveat.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, shell
- Domain
- performance, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100