Regression: [znver3] Loop unrolling regression for `_mm_dp_ps` / `vdpps` between LLVM 18 and LLVM 23
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
## Summary
With `-O2 -march=znver3`, LLVM 18 unrolls a throughput-bound `_mm_dp_ps` loop
**3× more aggressively** than LLVM 23, producing code that runs ~7× faster on
AMD Zen 3 hardware (measured on Ryzen 9 5900X).
Both compilers emit `vdpps` instructions; the performance difference is entirely
due to unroll depth. LLVM 18 splits the loop and unrolls the main body to
72 `vdpps` instructions; LLVM 23 does not split and unrolls to only 24.
---
## Reduced test case
```c
/* dpps_repro.c */
#include
/* 8 independent _mm_dp_ps accumulator chains — exposes throughput, not latency.
* On Zen 3: DPPS lat=11 cy, tp=4 cy. With 8 independent chains and sufficient
* unrolling the OoO engine can sustain near-peak throughput. */
__attribute__((noinline))
float dpps_throughput(const float *p, long n)
{
__m128 a0 = _mm_setzero_ps(), a1 = _mm_setzero_ps();
__m128 a2 = _mm_setzero_ps(), a3 = _mm_setzero_ps();
__m128 a4 = _mm_setzero_ps(), a5 = _mm_setzero_ps();
__m128 a6 = _mm_setzero_ps(), a7 = _mm_setzero_ps();
const __m128 *v = (const __m128 *)p;
for (long i = 0; i < n; i += 8) {
a0 = _mm_add_ps(a0, _mm_dp_ps(v[i+0], v[i+1], 0xFF));
a1 = _mm_add_ps(a1, _mm_dp_ps(v[i+2], v[i+3], 0xFF));
a2 = _mm_add_ps(a2, _mm_dp_ps(v[i+4], v[i+5], 0xFF));
a3 = _mm_add_ps(a3, _mm_dp_ps(v[i+6], v[i+7], 0xFF));
a4 = _mm_add_ps(a4, _mm_dp_ps(v[i+0], v[i+3], 0xFF));
a5 = _mm_add_ps(a5, _mm_dp_ps(v[i+2], v[i+5], 0xFF));
a6 = _mm_add_ps(a6, _mm_dp_ps(v[i+4], v[i+7], 0xFF));
a7 = _mm_add_ps(a7, _mm_dp_ps(v[i+6], v[i+1], 0xFF));
}
__m128 r = _mm_add_ps(_mm_add_ps(a0, a1), _mm_add_ps(a2, a3));
r = _mm_add_ps(r, _mm_add_ps(_mm_add_ps(a4, a5), _mm_add_ps(a6, a7)));
return _mm_cvtss_f32(r);
}
```
---
## How to reproduce
### Step 1 — compile and count `vdpps` in the loop body
```bash
clang-18 -O2 -march=znver3 -S -o dpps_clang18.s dpps_repro.c
clang-23 -O2 -march=znver3 -S -o dpps_clang23.s dpps_repro.c
grep -c vdpps dpps_clang18.s # 72
grep -c vdpps dpps_clang23.s # 24
```
### Step 2 — inspect LLVM IR loop metadata
```bash
clang-18 -O2 -march=znver3 -emit-llvm -S -o dpps_clang18.ll dpps_repro.c
clang-23 -O2 -march=znver3 -emit-llvm -S -o dpps_clang23.ll dpps_repro.c
grep "^!" dpps_clang18.ll
grep "^!" dpps_clang23.ll
```
### Step 3 — generate pre-optimized IR (for `opt` reproduction)
```bash
clang-18 -O1 -march=znver3 -Xclang -disable-llvm-passes \
-emit-llvm -S -o dpps_pre_opt.ll dpps_repro.c
```
This file can then be passed to `opt -O3 dpps_pre_opt.ll -S` to reproduce
the optimization difference without the front-end.
---
## Key evidence: IR loop metadata
### clang-18 (`Ubuntu clang version 18.1.3`)
```llvm
!8 = distinct !{!8, !9} ; loop 1 — cleanup loop
!9 = !{!"llvm.loop.unroll.disable"} ; ← unrolling explicitly disabled here
!10 = distinct !{!10, !11} ; loop 2 — main hot loop
!11 = !{!"llvm.loop.mustprogress"} ; no unroll constraint → backend unrolls aggressively
```
clang-18 **splits the loop into two**: a cleanup/tail loop (unroll disabled) and
a main hot loop (unconstrained). The backend then unrolls the main loop to
72 `vdpps` instructions for the znver3 target.
### clang-23 (`clang version 23.0.0git`)
```llvm
!9 = distinct !{!9, !10} ; single loop — no split performed
!10 = !{!"llvm.loop.mustprogress"} ; only mustprogress, no unroll metadata
```
clang-23 **does not split the loop**. A single loop body with no unroll
metadata is emitted; the backend unrolls to only 24 `vdpps` instructions,
reducing exploitable ILP.
---
## Measured performance (AMD Ryzen 9 5900X — Zen 3)
Measured with RDTSC serialized cycle counting, 31 samples, median reported,
8 independent accumulator chains:
```
Compiler / flags vdpps in loop cy/iter vs clang-18
───────────────────────────────── ───────────── ─────── ──────────
clang-18 -O2 -march=znver3 72 0.424 baseline
clang-23 -O2 -march=znver3 24 3.070 7.24× slower
clang-23 -O2 (no -march) 24 3.084 7.28× slower
gcc-13 -O2 -march=znver3 — 3.374 7.96× slower
```
The regression is specific to LLVM 18→23; all other compilers match the
clang-23 behaviour, suggesting the aggressive loop splitting was a
clang-18–era heuristic that was removed or changed.
---
## Why unroll depth matters here
`DPPS` / `vdpps` on Zen 3: latency = 11 cycles, throughput = 4 cycles (microcoded,
uses FP-add and FP-mul execution units).
With **72 `vdpps`** across 8 chains in the loop body, the out-of-order scheduler
(ROB size ~256 µops on Zen 3) can see all 8 chains simultaneously and dispatch a
new `vdpps` every 4 cycles, hiding the 11-cycle latency completely.
With only **24 `vdpps`** (3 per chain), the ROB window is refilled more frequently;
chains are not fully overlapped and the measured throughput degrades ~7×.
---
## Expected behaviour
`-O2 -march=znver3` should split the hot loop from the cleanup loop and unroll
the hot loop to a depth sufficient to keep the Zen 3 OoO engine fed, as
clang-18 did. The expected unroll depth for a tp=4, lat=11 instruction with 8
independent chains is ≥ 9× (72 instructions / 8 chains).
---
## Version information
- **Regressed version:** clang-23.0.0git (commit `68a9e9ca3e9324c48bdd541fdbc79e002ef7f2bb`)
- **Last known good:** Ubuntu clang 18.1.3 (`1ubuntu1`)
- **Bisect start:** LLVM 19 (first release after 18.x branch)
- **Source:** built from https://github.com/llvm/llvm-project.git
- **Target hardware:** AMD Ryzen 9 5900X (znver3), Linux x86-64
Contributor guide
Assessment
This issue has not been assessed yet.