ROCm / ROCm/aiter

[Issue]: Index Mismatch Between Main PA Kernel _paged_attention_kernel and Reduce Kernel _paged_attention_ll4mi_reduce_kernel

Open
#2,174 2 comments 0 reactions 1 assignee View on GitHub

@fsx950223 is already working on this.

Since Mar 16, 2026.

bug
Dominant language
Python
Stars
565
Forks
585
Avg merge
3d 4h
Merged PRs (30d)
366

Description

Problem Description

The paged_attention_v1 kernel API accepts an mtp parameter suggesting multi-token prediction support,
but values > 1 appear to be incorrect due to multiple implementation bugs.

The main kernel and reduce kernel use incompatible indexing schemes for the intermediate buffers (exp_sums, max_logits, tmp_out).

Main kernel _paged_attention_kernel writes to:
https://github.com/ROCm/aiter/blob/d2f5f2791336a9839c65241b77a88f0a8407a054/csrc/cpp_itfs/pa/pa_kernels.cuh#L660-L668

Reduce kernel_paged_attention_ll4mi_reduce_kernel reads from:
https://github.com/ROCm/aiter/blob/d2f5f2791336a9839c65241b77a88f0a8407a054/csrc/cpp_itfs/pa/pa_kernels.cuh#L926-L928

The indices don't match, meaning the reduce kernel reads incorrect data, silently producing wrong results.

Image
Operating System

Ubuntu 22.04.5 LTS (Jammy Jellyfish)

CPU

AMD EPYC 9575F 64-Core Processor

GPU

8 x AMD Instinct MI355X

ROCm Version

7.0.51831-a3e329ad8

ROCm Component

No response

Steps to Reproduce

Replicator script:
paged_attention_v1.issue1.mi355.py

Logs from running replicator script:
paged_attention_v1.issue1.log

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

My goal is to enable CUDA graph support for speculative decoding in the ROCM_AITER_FA attention backend by adding spec_as_decode support and changing _cudagraph_support from UNIFORM_SINGLE_TOKEN_DECODE to UNIFORM_BATCH. This is expected to speed up speculative decoding: removes graph breaks and allows routing speculative tokens through the more efficient decode path instead of the extend path.
The AITER PA API appears to support MTP>1, but the implementation seems to be buggy.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.