ROCm / ROCm/aiter

[Issue]: Potential Division by Zero in _paged_attention_kernel

Open
#2,175 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
565
Forks
585
Avg merge
3d 4h
Merged PRs (30d)
366

Description

Problem Description

https://github.com/ROCm/aiter/blob/d2f5f2791336a9839c65241b77a88f0a8407a054/csrc/cpp_itfs/pa/pa_kernels.cuh#L124-L125

The value of MTP_PARALLEL_THREADS is computed as:
https://github.com/ROCm/aiter/blob/d2f5f2791336a9839c65241b77a88f0a8407a054/csrc/cpp_itfs/pa/pa_kernels.cuh#L57-L60

If MTP_PARALLEL_THREADS > 4 (which can happen with large MTP values and certain GQA ratios),
then 4 / MTP_PARALLEL_THREADS = 0 due to integer division, causing division by zero and undefined behavior or a kernel crash.

Operating System

Ubuntu 22.04.5 LTS (Jammy Jellyfish)

CPU

AMD EPYC 9575F 64-Core Processor

GPU

8 x AMD Instinct MI355X

ROCm Version

ROCm version: 7.0.51831-a3e329ad8

ROCm Component

No response

Steps to Reproduce

Replicator script:
paged_attention_v1.issue2.mi355.py

Logs from running replicator script:
paged_attention_v1.issue2.mi355.log

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

My goal is to enable CUDA graph support for speculative decoding in the ROCM_AITER_FA attention backend by adding spec_as_decode support and changing _cudagraph_support from UNIFORM_SINGLE_TOKEN_DECODE to UNIFORM_BATCH. This is expected to speed up speculative decoding: removes graph breaks and allows routing speculative tokens through the more efficient decode path instead of the extend path. The AITER PA API appears to support MTP>1, but the implementation seems to be buggy.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with csrc/cpp_itfs/pa/pa_kernels.cuh, especially the MTP_PARALLEL_THREADS calculation at lines 57-60 and the division at lines 124-125. Run paged_attention_v1.issue2.mi355.py with its accompanying log, then verify the paged-attention kernel handles large MTP values and relevant GQA ratios without division-by-zero behavior or a crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.