mudler / mudler/vllm.cpp

Require distinct Q and K projections in native HRX Qwen fusion

Open
#3,089 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

The pinned AMD llama.cpp Qwen attention matcher accepts a K projection as both the query and key chain. When a complete graph visits K before Q, it emits one partial Qwen postprocess and leaves the real query to generic RMSNorm and RoPE. match_qwen_attention_postprocess in ggml/src/ggml-hrx/dispatch_registration/qwen/dispatch-qwen-attention-postprocess.cpp:623 does not require distinct query and key projection nodes.

The #3083 implementer owns the in-flow scheduling correction. The reservation helper already rejects this ambiguity, but the actual matcher must enforce the same complete-region invariant. Preserve the original matching and traversal defaults, numerical tolerances, and the complete Qwen fusion.

The CPU regression enters the public dispatch scheduler with BF16 Q16/KV8, token count1, and K-before-Q graph order. It fails at the assertion that a complete Qwen postprocess leaves no generic RMSNorm: exit -6, test ELF 1fe0b0bbf1a22a84fb652816ad943e678e415d167da8bfb9383c3db9afb613a8, unchanged backend fb2301563318df2253bc681272247a2a5587a42921a9b376b89fa103de70b2a9. Evidence: build-hrx-prefill-repair/evidence/key-first-scheduler-red.{json,log} in the #3083 worktree.

The #3083 spec will record this narrow matcher obligation before its implementation. A separate sequential GPU regression currently rejects NaN in the first decode K cache. This issue does not yet attribute that numerical failure to this matcher ambiguity; the operator will rerun it after the scoped correction.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in ggml/src/ggml-hrx/dispatch_registration/qwen/dispatch-qwen-attention-postprocess.cpp at match_qwen_attention_postprocess around line 623, then review the key-first scheduler evidence in build-hrx-prefill-repair/evidence/key-first-scheduler-red.{json,log}. Verify the matcher requires distinct query and key projection nodes while preserving the existing defaults and complete fusion, then rerun the CPU regression with BF16 Q16/KV8 and token count 1 to confirm no generic RMSNorm remains.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai-infra-agents, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.