mudler / mudler/vllm.cpp

Qualify native HRX Qwen prefill graph scheduling

Open
#3,083 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

The #3080 native HRX evaluation reaches Qwen3-0.6B generation after the separately tracked Loom repair (#3081), but longer prefill fails in the Qwen attention postprocess matcher.

Inputs: AMD-Ecosystem/llama.cpp 6319038132ed12f968ea68f37753f705da830ea8, HRX 6bcd5a4ff111fa5bf160ab9f4592ca8e7cc810b1 plus explicitly retained #3081 compiler patch, RX7900XTX gfx1100. Converted Qwen3-0.6B BF16 GGUF SHA256 2c7612c9a1d4f6d1f67fbee75ec88c089d1425696ac8361e9f73ca71023ea223. All 311 source tensors were checked against HF c1899de289a04d12100db370d81485cdf75e47ca.

Sixteen short prompts complete and match conventional HIP for all 256 output tokens. The retained 256-token prompt with context4096, batch512, microbatch512, F16 KV and flash attention enabled fails before generation: cannot cover value reshape node9, output f32[128,8,256,1]. The rejection originates in ggml/src/ggml-hrx/dispatch_registration/qwen/dispatch-qwen-attention-postprocess.cpp:693 through append_postprocess_node.

An explicit microbatch16 compatibility probe fails at the same value reshape, now f32[128,8,16,1]. Existing GGML_HRX_DISABLE_QWEN_DISPATCH=1 avoids that matcher but fails on unsupported per-head RMS_NORM f32[128,16,256,1]. These probes do not establish a hardware limitation or justify rejecting HRX. Standard HIP and its repaired HRX HIP-layer arm complete the same 256/2048-token requests and match all 512 generated tokens across the four qualification requests.

BACKEND-ROCM and the #3080 operator own diagnosis, a committed scoped spec before code, an upstream-native regression and minimal repair, fresh review and GPU qualification. Preserve the failed default configuration; do not silently replace it with disabled fusion or reduced context. The evaluation spec rocm-hrx-evaluation.md owns model/coverage qualification and will retain this issue under Owed in its final evidence update. Native broad prefill and model performance acceptance remain pending.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ggml/src/ggml-hrx/dispatch_registration/qwen/dispatch-qwen-attention-postprocess.cpp at append_postprocess_node around line 693, using the retained context4096, batch512, microbatch512 configuration and the reported value reshape failure. Compare the default path with the microbatch16 and disabled-dispatch probes, then add an upstream-native regression and minimal repair. Done means the default native HRX prefill qualifies without disabling fusion or reducing context.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.