mudler / mudler/vllm.cpp

gfx1151 survey: the vllm.cpp arm ran a PRE-#2511-fix binary, because survey.sh records our revision and never asserts it

Open
#2,933 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-GATE-ROCM-LLAMACPP

The multi-engine survey of 2026-09-04 (#2921) lost its whole vllm.cpp arm:
0 of 4 legs completed, every one rc=139 with HW Exception by GPU node-1 ... reason :GPU Hang, while llama.cpp and the pinned vLLM completed 4 of 4
each, interleaved, in the same lease on the same bytes.

The cause is not a new engine fault. The survey ran a binary built before the
fix for #2511 landed
, and nothing in the harness could notice.

The provenance, read off the run's own log

job.log records the revision and never checks it:

vllmcpp_source_revision=11fed3ba56b8f823c07032416982a44a8c0967b5
vllm_cli a703b83dd8954ba6dd3cbe82efcd38083c1d55492bbbaecf5c406f7c6efd646f

Both values are the ones #2511's own reproduction section names. That binary is
the pre-fix binary, byte for byte.

commit date (UTC) on origin/main
11fed3ba5 the surveyed revision 2026-09-01 16:15:47 yes
02ac7c5c2 put the managed branch behind a knob 2026-09-02 09:00:07 yes
6b97a6800 the fix: require PageableMemoryAccess 2026-09-02 11:38:23 yes

git merge-base --is-ancestor 11fed3ba5 6b97a6800 exits 0. The surveyed
revision is an ancestor of the fix, so the measured binary cannot contain
it. The survey ran on 2026-09-04, two days after the fix landed.

6b97a6800 measured the arm it repaired: hipMallocManaged failed 17 legs in
21
, plain hipMalloc 0 in 21. A 4-of-4 hang rate on the managed
allocator is what that commit describes, not a regression against it.

Why the harness could not catch it

survey.sh pins the oracle tree and refuses to run if it moved:

MAN=$(manifest_of "$LOCAL/src-llamacpp")
[ "$MAN" = "$LLAMA_MANIFEST" ] || fail "the staged oracle tree is not the pinned content"

For our own arm it checks that a file is executable and then prints the
revision:

[ -x "$VC" ] || fail "vllm.cpp binary absent"
echo "vllmcpp_source_revision=$(git -C "$LOCAL/src-vllmcpp" rev-parse HEAD ...)"

survey.sh never builds, never fetches and never asserts. /tmp/rocm-strix-q4k
is the #2511 campaign's own staging directory, so the survey silently inherited
that campaign's tree and build. A recorded value that no gate reads is a
comment. The engine under test is the one input this harness does not pin, and
it is the only one that failed.

Acceptance

  • survey.sh, and any sibling that stages our arm, asserts the vllm.cpp
    revision against a value the caller declares, and fails the run when it does
    not match. The same shape the llama.cpp manifest check already has.
  • The assertion covers the built bytes, not only the source revision. A source
    tree at the right commit with a stale build directory must also fail.
  • A red-before reproduction: the check refuses 11fed3ba5 when the run declares
    a post-6b97a6800 revision.

What this owns

The vllm.cpp row of the survey published for #2921 and #2497. That row reports
FAULTED, 0 of 4, rc 139 and carries no throughput figure. It is void as an
engine verdict
, because it measures a superseded revision, and it is published
as void rather than omitted.

Whether the current head completes this workload on gfx1151 is not
established by this run and is not claimed here
. It needs a fresh lease with
the revision asserted.

Reproduction

  • Device strix:gpu0, gfx1151, ROCm 7.2.4, boot id
    a5bc8128-f6ad-4767-8614-6923f88032e1, RC_JOB_ID
    b51afb48-ddb6-438a-b30c-acf46980e918.
  • Artifact Qwen3.8-27B-Q4_K_M.gguf, 17,106,775,008 B, sha256
    7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169, verified
    on the worker.
  • vllm-cli --model <gguf> --prompt 'The capital of France is' --max-tokens 64 --temperature 0 --repeat 4 --max-num-seqs 1, VT_OP_PROVIDER_STATS=1.
  • All four legs: reference_tier_notices=0, every op selected=vt-native. The
    hang is on the first forward pass, after 17 to 21 op-provider lines, roughly
    80 s into the leg. Board clocks during the faulted legs read 749 to 923 MHz
    mean and 6.1 to 12.2 percent busy, so the board never reached a working state.
  • Raw logs: /mnt/nas_share/rc/strix-survey-2497/out/survey-20260904/.

Related: #2511 (the fault and its fix), #2497, #2921.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with survey.sh, especially the existing llama.cpp manifest assertion and the vllm.cpp staging and revision logging. Trace how the built vllm.cpp bytes are selected, then reproduce the red-before case with 11fed3ba5 while declaring a post-6b97a6800 revision. Done means survey.sh and any sibling staging the arm reject a mismatched source revision or stale build, with the check covering the built bytes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, git, shell
Domain
build-system, testing-qa, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.