mudler / mudler/vllm.cpp

Qwen3.8-27B Q4_K_M: p4/14 is the CPU tier diverging across ARCHITECTURES, not ROCm; #2590 narrows to two steps

Open
#2,608 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: QUANT-QWEN38-27B-GGUF-ARM

#2590 was opened owning three steps at which our ROCm and CPU tiers emit a
different token from an identical prefix. Run on ONE host, in one lease, from one
libvllm.so, only two of them reproduce.

step recorded in #2590 measured on one host (strix:gpu0, rc job 5af7552a-5fba-4e51-92d8-f4cdfd3b21ca)
p1/45 ROCm 303, CPU 1521 ROCm 303, CPU 1521 — reproduces
p3/45 ROCm 25, CPU 393 ROCm 25, CPU 393 — reproduces
p4/14 ROCm 22486, CPU 4593 both tiers 22486; all 48 ids EQUAL

#2590's A vs D column compared a gfx1151 ROCm run on x86-64 against a
CPU-tier run on thor, aarch64, so it crossed an architecture as well as a
tier. With both arms on x86-64 our CPU tier emits 22486 at p4/14, agreeing with
the ROCm tier, with the HIP oracle and with the aarch64 oracle.

So p4/14 is the CPU tier's own architecture dependence, not a ROCm-local
term.
The token the aarch64 CPU tier emitted there (4593) is one no other
side emits. This issue owns that, and #2590 is narrowed to two steps.

What is already known

  • The step is a near-tie: the HIP oracle's top1 - top2 gap at p4/14 is
    0.092752, inside the 19-step population whose gap is below 0.20.
  • The x86-64 CPU tier is bit-reproducible per layer against itself, and so is the
    ROCm tier: the run-to-run floor measured here is exactly 0 over 9360 compared
    per-layer rows.
  • The cross-tier hidden-state divergence at layer 63 is 2.6-2.8% relative, which
    is 1.03 to 1.11 times what two independent bf16 residual streams must produce.
    A near-tie under roughly half a nat is decided by that, so a divergence at a
    0.0928 gap needs no further cause than a different reduction order.
  • The one candidate that is NOT explained by that is why aarch64 and x86-64
    differ from each other, which is what this issue is for. QuantRepackForDevice
    is dev == kCPU only and the aarch64 lane carries the block_q8_0x4 interleave
    (#2406); x86-64 does not repack Q4_K either, so the repack is a candidate and
    not a conclusion.

The next traceable step

Run the same per-layer hidden-state comparison this measurement built
(scripts/tier-hidden-delta.py, scripts/tier-delta-attribution.py,
VT_DUMP_ACT) with the two arms being aarch64 CPU and x86-64 CPU on the same
artifact and the same prompt, and read the per-layer profile the same way. The
instrument is committed and needs no GPU on either side.

Not admissible from this

No speed, latency or memory number. The arm's declared token gate is FAIL and
AGENTS.md §Gates admits none from it.

Evidence:
docs/bench-evidence/qwen38-27b-q4km-tier-divergence-bisect-20260902.md.
Spec: .agents/specs/rocm-tier-hidden-state-bisect.md.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-1m [claude-code]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/tier-hidden-delta.py and scripts/tier-delta-attribution.py, using VT_DUMP_ACT with the same artifact and prompt on aarch64 CPU and x86-64 CPU. Compare the per-layer hidden-state profiles and trace where the architectures diverge; done means documenting the attributable divergence and whether the block_q8_0x4 repack remains a candidate.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.