mudler / mudler/vllm.cpp

llama.cpp b10451 built for HIP emits degenerate output at n_gpu_layers=0 on gfx1151's host, while n_gpu_layers=99 is correct

Open
#2,557 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

Measured 2026-09-02 on strix:gpu0 (rc job 2644106f-3123-4c18-8d42-d4db11fe62e8,
worker rc-worker-lcjhd, x86-64, AMD RYZEN AI MAX+ 395) while running the
gfx1151 Q4_K_M token gate for
#2546.

One stock b10451 binary, built with -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151,
loaded the recorded Qwen3.8-27B-Q4_K_M.gguf (sha256
7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169) twice in one
job, greedy, 48 tokens, identical prompts. The only difference is
n_gpu_layers.

n_gpu_layers = 99 is coherent and correct. Its ids agree with the
2026-08-23 aarch64 run of the same oracle on 3 of 6 prompts exactly and differ on
3 by one token each.

n_gpu_layers = 0 is degenerate. It emits one repeated token for the whole
48-token completion on 5 of 6 prompts:

prompt 0: n=48 distinct=1  first5=[286, 286, 286, 286, 286]
prompt 1: n=48 distinct=3  first5=[90, 13, 641, 641, 641]
prompt 2: n=48 distinct=1  first5=[383, 383, 383, 383, 383]
prompt 3: n=48 distinct=1  first5=[393, 393, 393, 393, 393]
prompt 4: n=48 distinct=1  first5=[96229, 96229, ...]
prompt 5: n=48 distinct=1  first5=[12512, 12512, ...]

system_info for both legs:
ROCm : NO_VMM = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1,
use_extra_bufts = 1, devices ROCm0 Radeon 8060S Graphics and
CPU AMD RYZEN AI MAX+ 395 w/ Radeon 8060S.

Why this matters beyond a broken leg

That leg existed to be a bridge: the CPU tier's recorded oracle ids came from
aarch64, so an x86-64 CPU leg from the same build was meant to separate the host
term from the kernel-path term when comparing tiers. It cannot serve that purpose
and was not used. The comparison in #2546 rests on the HIP leg against the
recorded aarch64 leg instead.

It also means no CPU-tier number should be taken from a HIP-built llama.cpp on
this host
until this is explained.

Not yet discriminated

This is recorded as an observation, not a diagnosis. The harness is the
2026-08-23 oracle_tokens.cpp, unchanged except for reading n_gpu_layers and
use_extra_bufts from the environment, and the identical code path is coherent
at ngl=99, so the harness is not obviously implicated. Three candidates, in the
order they are cheap to test:

  1. use_extra_bufts=1 (the AVX512 Q4_K repack path) on this CPU. Re-run the same
    leg with ORACLE_EXTRA_BUFTS=0; if it is coherent, the repack path is the
    cause.
  2. A mixed-backend scheduling defect at ngl=0 in a HIP-enabled build, where the
    ROCm backend is registered but no layer is offloaded. Test with a CPU-only
    build of the same pin on the same host.
  3. Board or driver state left behind by the preceding HIP leg in the same
    process lifetime. Test by running the ngl=0 leg first in a fresh process.

Whoever picks this up should run 1 and 2 before anything else; they are one lease
between them and they partition the space.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the unchanged 2026-08-23 oracle_tokens.cpp harness and reproduce the gfx1151 ngl=0 leg. Run the two requested checks first: repeat with ORACLE_EXTRA_BUFTS=0, then compare a CPU-only build on the same host. Done means identifying which candidate explains the degenerate output and recording whether the coherent ngl=99 behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.