gfx1151 survey: the vllm.cpp arm ran a PRE-#2511-fix binary, because survey.sh records our revision and never asserts it
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-GATE-ROCM-LLAMACPP
The multi-engine survey of 2026-09-04 (#2921) lost its whole vllm.cpp arm:
0 of 4 legs completed, every one rc=139 with HW Exception by GPU node-1 ... reason :GPU Hang, while llama.cpp and the pinned vLLM completed 4 of 4
each, interleaved, in the same lease on the same bytes.
The cause is not a new engine fault. The survey ran a binary built before the
fix for #2511 landed, and nothing in the harness could notice.
The provenance, read off the run's own log
job.log records the revision and never checks it:
vllmcpp_source_revision=11fed3ba56b8f823c07032416982a44a8c0967b5
vllm_cli a703b83dd8954ba6dd3cbe82efcd38083c1d55492bbbaecf5c406f7c6efd646f
Both values are the ones #2511's own reproduction section names. That binary is
the pre-fix binary, byte for byte.
| commit | date (UTC) | on origin/main |
|---|---|---|
11fed3ba5 the surveyed revision |
2026-09-01 16:15:47 | yes |
02ac7c5c2 put the managed branch behind a knob |
2026-09-02 09:00:07 | yes |
6b97a6800 the fix: require PageableMemoryAccess |
2026-09-02 11:38:23 | yes |
git merge-base --is-ancestor 11fed3ba5 6b97a6800 exits 0. The surveyed
revision is an ancestor of the fix, so the measured binary cannot contain
it. The survey ran on 2026-09-04, two days after the fix landed.
6b97a6800 measured the arm it repaired: hipMallocManaged failed 17 legs in
21, plain hipMalloc 0 in 21. A 4-of-4 hang rate on the managed
allocator is what that commit describes, not a regression against it.
Why the harness could not catch it
survey.sh pins the oracle tree and refuses to run if it moved:
MAN=$(manifest_of "$LOCAL/src-llamacpp")
[ "$MAN" = "$LLAMA_MANIFEST" ] || fail "the staged oracle tree is not the pinned content"
For our own arm it checks that a file is executable and then prints the
revision:
[ -x "$VC" ] || fail "vllm.cpp binary absent"
echo "vllmcpp_source_revision=$(git -C "$LOCAL/src-vllmcpp" rev-parse HEAD ...)"
survey.sh never builds, never fetches and never asserts. /tmp/rocm-strix-q4k
is the #2511 campaign's own staging directory, so the survey silently inherited
that campaign's tree and build. A recorded value that no gate reads is a
comment. The engine under test is the one input this harness does not pin, and
it is the only one that failed.
Acceptance
survey.sh, and any sibling that stages our arm, asserts the vllm.cpp
revision against a value the caller declares, and fails the run when it does
not match. The same shape the llama.cpp manifest check already has.- The assertion covers the built bytes, not only the source revision. A source
tree at the right commit with a stale build directory must also fail. - A red-before reproduction: the check refuses
11fed3ba5when the run declares
a post-6b97a6800revision.
What this owns
The vllm.cpp row of the survey published for #2921 and #2497. That row reports
FAULTED, 0 of 4, rc 139 and carries no throughput figure. It is void as an
engine verdict, because it measures a superseded revision, and it is published
as void rather than omitted.
Whether the current head completes this workload on gfx1151 is not
established by this run and is not claimed here. It needs a fresh lease with
the revision asserted.
Reproduction
- Device
strix:gpu0,gfx1151, ROCm 7.2.4, boot id
a5bc8128-f6ad-4767-8614-6923f88032e1,RC_JOB_ID
b51afb48-ddb6-438a-b30c-acf46980e918. - Artifact
Qwen3.8-27B-Q4_K_M.gguf, 17,106,775,008 B, sha256
7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169, verified
on the worker. vllm-cli --model <gguf> --prompt 'The capital of France is' --max-tokens 64 --temperature 0 --repeat 4 --max-num-seqs 1,VT_OP_PROVIDER_STATS=1.- All four legs:
reference_tier_notices=0, every opselected=vt-native. The
hang is on the first forward pass, after 17 to 21 op-provider lines, roughly
80 s into the leg. Board clocks during the faulted legs read 749 to 923 MHz
mean and 6.1 to 12.2 percent busy, so the board never reached a working state. - Raw logs:
/mnt/nas_share/rc/strix-survey-2497/out/survey-20260904/.
Related: #2511 (the fault and its fix), #2497, #2921.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with survey.sh, especially the existing llama.cpp manifest assertion and the vllm.cpp staging and revision logging. Trace how the built vllm.cpp bytes are selected, then reproduce the red-before case with 11fed3ba5 while declaring a post-6b97a6800 revision. Done means survey.sh and any sibling staging the arm reject a mismatched source revision or stale build, with the check covering the built bytes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, git, shell
- Domain
- build-system, testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100