mudler / mudler/vllm.cpp

fix(BACKEND-ROCM): bound HRX decode writes to each KV group

Open
#3,091 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

The native HRX cooperative split-K attention reducer permits multiple KV groups to write the same query-head output. At AMD-Ecosystem/llama.cpp pin 6319038132ed12f968ea68f37753f705da830ea8, flash_attention_decode_split_f32_f16_wmma.loom:578 tests each global head against the total query-head count. It must test the local query row against the heads owned by that KV group.

For Qwen3-0.6B with 16 query heads and 8 KV heads, KV0 can write heads 0–7 and KV1 can write heads 2–9. The cooperative path begins at KV capacity 257. The unchanged direct reducer owns exactly two heads per KV group.

The #3083 sequential backend fixture fails on its first decode at position 256: maximum absolute error 0.056254 exceeds the unchanged upstream flash-attention bound of 0.05. Shared weights are byte-identical and the cache-only control passes. The operator's whole-model boundary test matches HIP for all 128 output IDs after prompts of length 16 and 128. Prompt 240 first diverges at output index 17; prompt 256 first diverges at index 1. Both failures begin at decode position 256.

The #3083 implementer owns the bounded validity-predicate repair in the same flow. Preserve dispatch, arithmetic, thresholds, the #3081 provider, and upstream pins. Retain the numerical and whole-model red evidence, then require green and a fresh mutation review. Parent: #3080. Spec: .agents/specs/rocm-hrx-prefill-repair-3083.md.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with flash_attention_decode_split_f32_f16_wmma.loom at line 578 and read .agents/specs/rocm-hrx-prefill-repair-3083.md alongside the #3083 sequential backend fixture. Verify the local query-row validity bounds each KV group without changing dispatch, arithmetic, thresholds, provider #3081, or upstream pins. Done means the numerical and whole-model checks are green and a fresh mutation review is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.