mudler / mudler/vllm.cpp

gfx1151: the vllm.cpp survey arm has never been measured on a post-#2511 head, so its fault rate is unknown

Open
#2,944 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-GATE-ROCM-LLAMACPP

The gfx1151 multi-engine survey published for #2921 reports the vllm.cpp arm as
FAULTED, 0 of 4, rc 139, and #2933 established why: the run measured
11fed3ba5, a revision that predates the fix for #2511. survey.sh asserted the
llama.cpp source manifest and refused to run when it moved, but for our own arm
it only checked [ -x "$VC" ] and printed the revision. It never built,
never fetched and never asserted, so it inherited #2511's staging directory at
/tmp/rocm-strix-q4k and measured that campaign's pre-fix build.

So the published fault rate is #2511 reproducing, not a regression.

What is still unmeasured

Whether the current head completes this workload on gfx1151 is not
established by that run and is not established by #2511 either. #2511's
"0 faults in 21 legs" was a different workload (--max-tokens 288 --repeat 1
and --max-tokens 64 --repeat 4 against a branch build) in a different lease.
Neither answers the survey's own question.

What this owns

One lease on strix:gpu0 that measures the vllm.cpp arm of the #2921 survey on
a binary built from main's head inside that lease, with the revision
asserted rather than printed, and reports the fault rate honestly whatever
it is.

Acceptance

  • The job clones the declared revision, asserts rev-parse HEAD and
    rev-parse HEAD^{tree} against values the caller declares, asserts the
    worktree is unmodified, and exits non-zero when any of them disagrees.
    Fail-closed is the whole point: #2933 exists because the check was a printf.
  • The build directory is removed before configure, so a stale build cannot be
    inherited the way /tmp/rocm-strix-q4k was. The built libvllm.so and
    vllm-cli sha256 are recorded, and the run refuses the exact pre-fix
    vllm-cli sha256 a703b83dd8954ba6dd3cbe82efcd38083c1d55492bbbaecf5c406f7c6efd646f
    that #2933 names.
  • The workload is the survey's own: Qwen3.8-27B-Q4_K_M.gguf, sha256
    7e78da5d…fe169, 17,106,775,008 B verified on the worker,
    --prompt 'The capital of France is' --max-tokens 64 --temperature 0 --repeat 4 --max-num-seqs 1, four interleaved rounds, run 1 of each leg cold
    and discarded, an AMD clock window sampled beside every leg, and N taken from
    the design rather than from a log tally.
  • A second token count (128) is timed in the same lease, so a decode-only figure
    can be derived the same way the survey derived vLLM's, and labelled
    derived wherever it appears. Without it our figure is whole-completion and
    cannot be divided by llama.cpp's llama-bench -p 0 decode number: PR #2940
    records that exact mismatch emitting a meaningless 1.814x.
  • The fault rate is published whatever it is. If the current head still faults,
    that supersedes #2933's diagnosis and is reported as the result.
  • The correctness caveat travels with the number, on the face of the table:
    TOKEN_GATE=FAIL, every divergence a near-tie at about 0.125 nats, no
    deterministic denominator on this path. Those divergence counts are constants
    carried from earlier evidence and are not re-measured here.

Not in scope

Re-measuring llama.cpp or vLLM. Their figures stand at 12.233 tok/s
(docs/bench-evidence/rocm-strix-llamacpp-denominator-20260902.md), 12.219 from
the survey's own legs, and 6.734 tok/s whole completion / 11.056 derived decode.
Repairing survey.sh itself, which is #2933's own acceptance.

Related: #2933, #2497, #2921, #2511, #2940.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the BACKEND-GATE-ROCM-LLAMACPP survey row and the existing vllm.cpp job entry point, including survey.sh for its current revision checks. Run the workload on strix:gpu0 with a clean build directory, verify the declared Git revision, tree, worktree, binaries, model, clock window, and token counts, then publish the fault rate and derived decode figure with the TOKEN_GATE=FAIL caveat.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, shell
Domain
performance, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.