[Feature]: Support tp=4 when number of heads = 32 for Deepseek MLA aiter kernel
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 565
- Forks
- 585
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 366
Description
A community user is able to fit Deepseek R1 model on 4 GPUs on MI325x.
We were able to recreate the issue on MI300x.
When aiter MLA kernel is enabled, it would throw exception
Example error log in vLLM V0:
[rank0]: File "/app/vllm/vllm/attention/ops/rocm_aiter_mla.py", line 67, in mla_decode_fwd_impl
[rank0]: mla_decode_fwd(q,
[rank0]: File "/app/aiter/aiter/mla.py", line 116, in mla_decode_fwd
[rank0]: num_kv_splits, mgc = get_meta_param(num_kv_splits, device, bs, nhead)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/app/aiter/aiter/mla.py", line 89, in get_meta_param
[rank0]: assert nhead in get_mgc, f"{nhead=} not supported"
[rank0]: ^^^^^^^^^^^^^^^^
[rank0]: AssertionError: nhead=32 not supported
Example error on vLLM V1:
AssertionError: Aiter MLA only supports 16 or 128 number of heads.
Provided 32 number of heads.
Try adjusting tensor_parallel_size value.
Checked the aiter source code, assertion in source code :
https://github.com/ROCm/aiter/blob/main/aiter/mla.py#L159
Note:
This issue is created to follow up on vLLM issue: https://github.com/vllm-project/vllm/issues/19332.
Operating System
No response
GPU
MI300 series
ROCm Component
AITER MLA kernel
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at aiter/mla.py line 159, where the assertion rejects 32 heads, and trace the metadata path through get_meta_param and mla_decode_fwd. Reproduce the DeepSeek MLA case with tensor parallelism 4 on the documented MI300-series setup; done means the 32-head configuration no longer raises the unsupported-head assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100