llvm / llvm/llvm-project

[AMDGPU] Packed-integer codegen performance gap vs RADV/ACO on gfx1151 (3.2x slower)

Open
#219,248 3 comments 0 reactions 0 assignees View on GitHub
backend:AMDGPU performance
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

LLVM AMDGPU backend generates significantly slower code than RADV/ACO for packed-integer operations on gfx1151, despite emitting identical instructions. This is **Issue #3** (LLVM codegen) split from parent issue [ROCm/ROCm#6409.](https://github.com/ROCm/legacy-rocm-build/issues/6409).

**Performance Gap:**
- **gfx1151 (Strix Halo):** Vulkan 3.05x-3.20x faster than HIP
- **gfx1100 (W7900):** Vulkan 1.05x-1.13x faster than HIP (minor gap)

## Evidence

Serialized workgroup-64 examples on gfx1151:

| Variant | HIP GPU time (μs) | Vulkan GPU time (μs) | Speedup | HIP dot4 count | RADV dot4 count |
|---------|-------------------|----------------------|---------|----------------|-----------------|
| q8 signed | 3581.117 | 1121.940 | **3.192x** | 16 | 16 |
| q4 unsigned × signed-q8 | 3575.964 | 1119.920 | **3.193x** | 16 | 16 |
| q6 zero-corrected | 3429.718 | 1122.946 | **3.054x** | 32 | 32 |
| **scalar q4 dequant (NO dot4)** | 3661.869 | 1142.739 | **3.204x** | **0** | **0** |

**Key observation:** Both backends emit the expected dot4 instructions. The scalar control emits **zero dot4 instructions** on both sides, yet gfx1151 still shows **3.204x** gap. This gap is **not** due to missing dot4 lowering—it's a **codegen quality issue** (scheduling, waits, address generation, occupancy).

## Architecture Dependency

- **gfx1151 (RDNA3.5):** 3.05x-3.20x gap (MAJOR)
- **gfx1100 (RDNA3):** 1.05x-1.13x gap (minor)

This is a genuine device-dependent codegen difference, not a version mismatch. Both devices use the **identical software stack**.

## Test Configuration

| Component | Version |
|-----------|---------|
| ROCm | TheRock 7.15.0a20260711 |
| AMD clang/LLVM | 23.0.0git, source `aa451e1f`, patch `440716f8` |
| Mesa/RADV | 26.1.4-arch3.1 |
| Kernel | linux-cachyos 7.1.3-2 |
| Firmware | linux-firmware 1:20260622-1 |

**Devices:**
- gfx1151: Radeon 8060S / Strix Halo (40 CUs)
- gfx1100: Radeon Pro W7900 (96 CUs)

**Workload parameters:**
- n=32768, body_iters=64, 16 packed groups/iter
- Workgroups: 64/256, samples: 10/3/5

**Wave configuration:**
- HIP: wave32
- Vulkan: wave64
- Programs are **operationally matched**, not textually identical

## Reproduction

Clone the benchmark suite:

```bash
git clone https://github.com/shisa-ai/hipEngine.git
cd hipEngine

# gfx1151 measured source
git checkout 0e566a4559b52a8bfc65ccdbda22556ae9112279

# Set up environment
bash scripts/update-therock-torch.sh \
--device gfx1151 --date 20260711 --test --verify-torch

export HIPENGINE_HIP_ARCH=gfx1151
export GPU_NAME="Radeon 8060S Graphics"
```

Run packed-integer benchmark (see [command templates](https://github.com/shisa-ai/hipEngine/blob/main/benchmarks/micro/README.md#current-gfx1151-command-templates)).

**Retained artifact:** [gfx1151 results JSON](https://github.com/shisa-ai/hipEngine/blob/f2c3ad6d74c86e3641ce09ff9fd759eaa6cd75e0/benchmarks/results/2026-07-11-gfx1151-hip-vulkan-matched-protocol.json)

## Requested Investigation

1. **Which AMD clang/LLVM team owns gfx11 packed-integer lowering and scheduling?**
2. **What artifacts are needed?**
- Preprocessed source
- LLVM IR/bitcode
- HSACO/disassembly
- Occupancy analysis
- Wait state analysis
- Issue counters
3. **Is there a preferred HIP builtin or source form for these loops?**

## Related Issues

**Parent issue:** [ROCm/ROCm#6409 - HIP/LLVM performance gaps vs RADV/Vulkan (3-part split)](https://github.com/ROCm/legacy-rocm-build/issues/6409)
- **Issue A (HIP Runtime):** Graph replay overhead (10x slower) https://github.com/ROCm/rocm-systems/issues/10834
- **Issue B (HIP Runtime):** Stream concurrency gap (21x slower on gfx1151) https://github.com/ROCm/rocm-systems/issues/10836
- **Issue C (LLVM - THIS ISSUE):** Packed-integer codegen performance

Contributor guide

Open the contributing guide

Research direction

Start with the pinned hipEngine revision, the gfx1151 command templates in benchmarks/micro/README.md, and the retained results JSON. Reproduce the benchmark, then collect the requested preprocessed source, LLVM IR, HSACO/disassembly, occupancy, wait-state, and issue-counter artifacts. Done means identifying the AMDGPU codegen cause of the gap and documenting the relevant source form or investigation result.

Written by the indexing model from the issue text.

Assessment

Domain
compilers, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.