vllm-project / vllm-project/vllm

[ROCm][AMD] GLM5.2/5.3 Performance Optimization on gfx950 / MI355X

Open
#57,230 2 comments 1 reaction 0 assignees View on GitHub
feature request quantization rocm
Dominant language
Python
Stars
92k
Forks
22.3k
PR merge metrics
PR metrics pending

Description

This feature page tracks the peformance optimization of [amd/GLM-5.2-MXFP4](https://huggingface.co/amd/GLM-5.2-MXFP4) / [amd/GLM-5.3-Quark-MXFP4-AttnFP8](https://huggingface.co/amd/GLM-5.3-Quark-MXFP4-AttnFP8) on gf950/MI355X
The following list of PRs lands as stacked PRs on vllm-project/vllm main.

CC:
@tjtanaa, @jhartika-amd, @maeehart, @nholmber

Nvidia equivalence
- [In review] Index Share feature (1% E2E TTFT improvement) - vllm-project/vllm#51309
- [In review] Remove aten fill functor in prefill MQA logits (1% E2E TTFT improvement) - vllm-project/vllm#51314

AITER kernel optimization:
- [In review] use AITER kernel for `cp_gather_indexer_k_quant_cache` and `indexer_k_quant_and_cache` and `top_k_per_row_decode` instead of triton - vllm-project/vllm#46172
- [In review] Sparse indexer kernel fusion for K normalization, Q/K RoPE, FP8 quantization, and K-cache write - `indexer_qk_rope_quant_and_cache` (1% E2E throughput improvement) - vllm-project/vllm#51315
- [In review] MLA fused kernel for rope, query concat, KV concat, fp8 cache write -` fused_qk_rope_concat_and_cache_mla` - vllm-project/vllm#47757
- [In review] Replace ATen gemm with tuned AITER gemm `aiter.tuned_gemm.tgemm.mm` (3% E2E throughput improvement) - vllm-project/vllm#50535
- [closed] fMoE tuning for gfx950 (5% E2E throughput improvement) - ROCm/aiter#4629
- [open in aiter] Dispatch to FlyDSL MQA logits in prefill `flydsl_fp8_mqa_logits` (1.5% E2E throughput improvement)

Additional optimization
- [ready] FP8 dtype hoist - vllm-project/vllm#53792

AITER dependencies that need to be merged
- FlyDSL paged MQA logits in decode - ROCm/aiter#4221
- FlyDSL MQA logits in prefill - ROCm/aiter#4538
- FlyDSL topk - ROCm/aiter#4355

Planned tasks
- [open in aiter] Dispatch to FlyDSL paged MQA logits in decode `flydsl_fp8_paged_mqa_logits` (1 week ETA, expected 1.5% E2E throughput improvement)
- MTP enablement:
- support next_n > 2
- AITER kernel support
- FlyDSL kernel for quick all reduce (INT4) - ROCm/aiter#4970​

Contributor guide

Open the contributing guide

Research direction

This issue is a tracking page rather than a single change. Start by reviewing the listed vLLM and ROCm/aiter PRs and planned tasks, then identify one unclaimed optimization with a defined benchmark target. Done means the selected task and its required dependency are implemented, merged, and show the stated performance improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai-infra-agents, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.