vllm-project / vllm-project/vllm
[ROCm][AMD] GLM5.2/5.3 Performance Optimization on gfx950 / MI355X
- Dominant language
- Python
- Stars
- 92k
- Forks
- 22.3k
- PR merge metrics
- PR metrics pending
Description
This feature page tracks the peformance optimization of [amd/GLM-5.2-MXFP4](https://huggingface.co/amd/GLM-5.2-MXFP4) / [amd/GLM-5.3-Quark-MXFP4-AttnFP8](https://huggingface.co/amd/GLM-5.3-Quark-MXFP4-AttnFP8) on gf950/MI355X
The following list of PRs lands as stacked PRs on vllm-project/vllm main.
CC:
@tjtanaa, @jhartika-amd, @maeehart, @nholmber
Nvidia equivalence
- [In review] Index Share feature (1% E2E TTFT improvement) - vllm-project/vllm#51309
- [In review] Remove aten fill functor in prefill MQA logits (1% E2E TTFT improvement) - vllm-project/vllm#51314
AITER kernel optimization:
- [In review] use AITER kernel for `cp_gather_indexer_k_quant_cache` and `indexer_k_quant_and_cache` and `top_k_per_row_decode` instead of triton - vllm-project/vllm#46172
- [In review] Sparse indexer kernel fusion for K normalization, Q/K RoPE, FP8 quantization, and K-cache write - `indexer_qk_rope_quant_and_cache` (1% E2E throughput improvement) - vllm-project/vllm#51315
- [In review] MLA fused kernel for rope, query concat, KV concat, fp8 cache write -` fused_qk_rope_concat_and_cache_mla` - vllm-project/vllm#47757
- [In review] Replace ATen gemm with tuned AITER gemm `aiter.tuned_gemm.tgemm.mm` (3% E2E throughput improvement) - vllm-project/vllm#50535
- [closed] fMoE tuning for gfx950 (5% E2E throughput improvement) - ROCm/aiter#4629
- [open in aiter] Dispatch to FlyDSL MQA logits in prefill `flydsl_fp8_mqa_logits` (1.5% E2E throughput improvement)
Additional optimization
- [ready] FP8 dtype hoist - vllm-project/vllm#53792
AITER dependencies that need to be merged
- FlyDSL paged MQA logits in decode - ROCm/aiter#4221
- FlyDSL MQA logits in prefill - ROCm/aiter#4538
- FlyDSL topk - ROCm/aiter#4355
Planned tasks
- [open in aiter] Dispatch to FlyDSL paged MQA logits in decode `flydsl_fp8_paged_mqa_logits` (1 week ETA, expected 1.5% E2E throughput improvement)
- MTP enablement:
- support next_n > 2
- AITER kernel support
- FlyDSL kernel for quick all reduce (INT4) - ROCm/aiter#4970
Contributor guide
Research direction
This issue is a tracking page rather than a single change. Start by reviewing the listed vLLM and ROCm/aiter PRs and planned tasks, then identify one unclaimed optimization with a defined benchmark target. Done means the selected task and its required dependency are implemented, merged, and show the stated performance improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai-infra-agents, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100