ROCm / ROCm/aiter

[Feature]: Support tp=4 when number of heads = 32 for Deepseek MLA aiter kernel

Open
#548 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
565
Forks
585
Avg merge
3d 4h
Merged PRs (30d)
366

Description

A community user is able to fit Deepseek R1 model on 4 GPUs on MI325x.

We were able to recreate the issue on MI300x.

When aiter MLA kernel is enabled, it would throw exception

Example error log in vLLM V0:

[rank0]:   File "/app/vllm/vllm/attention/ops/rocm_aiter_mla.py", line 67, in mla_decode_fwd_impl
[rank0]:     mla_decode_fwd(q,
[rank0]:   File "/app/aiter/aiter/mla.py", line 116, in mla_decode_fwd
[rank0]:     num_kv_splits, mgc = get_meta_param(num_kv_splits, device, bs, nhead)
[rank0]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/app/aiter/aiter/mla.py", line 89, in get_meta_param
[rank0]:     assert nhead in get_mgc, f"{nhead=} not supported"
[rank0]:            ^^^^^^^^^^^^^^^^
[rank0]: AssertionError: nhead=32 not supported

Example error on vLLM V1:

AssertionError: Aiter MLA only supports 16 or 128 number of heads.
Provided 32 number of heads.
Try adjusting tensor_parallel_size value.

Checked the aiter source code, assertion in source code :
https://github.com/ROCm/aiter/blob/main/aiter/mla.py#L159

Note:
This issue is created to follow up on vLLM issue: https://github.com/vllm-project/vllm/issues/19332.

Operating System

No response

GPU

MI300 series

ROCm Component

AITER MLA kernel

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at aiter/mla.py line 159, where the assertion rejects 32 heads, and trace the metadata path through get_meta_param and mla_decode_fwd. Reproduce the DeepSeek MLA case with tensor parallelism 4 on the documented MI300-series setup; done means the 32-head configuration no longer raises the unsupported-head assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.