[Issue]: Potential Division by Zero in _paged_attention_kernel
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 565
- Forks
- 585
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 366
Description
Problem Description
The value of MTP_PARALLEL_THREADS is computed as:
https://github.com/ROCm/aiter/blob/d2f5f2791336a9839c65241b77a88f0a8407a054/csrc/cpp_itfs/pa/pa_kernels.cuh#L57-L60
If MTP_PARALLEL_THREADS > 4 (which can happen with large MTP values and certain GQA ratios),
then 4 / MTP_PARALLEL_THREADS = 0 due to integer division, causing division by zero and undefined behavior or a kernel crash.
Operating System
Ubuntu 22.04.5 LTS (Jammy Jellyfish)
CPU
AMD EPYC 9575F 64-Core Processor
GPU
8 x AMD Instinct MI355X
ROCm Version
ROCm version: 7.0.51831-a3e329ad8
ROCm Component
No response
Steps to Reproduce
Replicator script:
paged_attention_v1.issue2.mi355.py
Logs from running replicator script:
paged_attention_v1.issue2.mi355.log
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
My goal is to enable CUDA graph support for speculative decoding in the ROCM_AITER_FA attention backend by adding spec_as_decode support and changing _cudagraph_support from UNIFORM_SINGLE_TOKEN_DECODE to UNIFORM_BATCH. This is expected to speed up speculative decoding: removes graph breaks and allows routing speculative tokens through the more efficient decode path instead of the extend path. The AITER PA API appears to support MTP>1, but the implementation seems to be buggy.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with csrc/cpp_itfs/pa/pa_kernels.cuh, especially the MTP_PARALLEL_THREADS calculation at lines 57-60 and the division at lines 124-125. Run paged_attention_v1.issue2.mi355.py with its accompanying log, then verify the paged-attention kernel handles large MTP values and relevant GQA ratios without division-by-zero behavior or a crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100