[AArch64][SVE2] clang 24 trunk generates ~1.5x slower code than GCC 16 for Highway's VQSort sorting networks at the SVE2_128 target — low IPC, backend-bound
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
Same source, same Highway target, same machine, clang is 1.53x slower than GCC.
```
clang 24.0.0git target=SVE2_128 VQSort(double, 2M) median 45.31 ms
g++ 16.1.1 target=SVE2_128 VQSort(double, 2M) median 29.58 ms
```
Both compilers dispatch to the same Highway target (`SVE2_128`), so this is not target selection or
feature detection. Seems to be codegen.
The `-mcpu` flag does not move it:
```
clang 24.0.0git -mcpu=gb10 SVE2_128 45.31 ms
clang 24.0.0git -mcpu=native SVE2_128 45.56 ms
clang 24.0.0git -mcpu=gb10+nosve SVE2_128 46.20 ms
g++ 16.1.1 -mcpu=native SVE2_128 29.58 ms
```
`+nosve` not moving the number is interesting. clang lands on the same SVE2_128 Highway target either
way and is equally slow.
Highway's VQSort is what NumPy's `np.sort` dispatches into, and NumPy vendors this Highway tree
verbatim. Same NumPy version on the same box, built two ways:
```
np.sort(2M float64) PyPI wheel, built by GCC 14.2.1 30.8 ms
np.sort(2M float64) built from sdist by clang 24 47.7 ms
```
`np.unique` is 1.53x slower for the same reason, it sorts internally. `np.argsort` and `mergesort` are
unaffected, and `dgemm` is exactly equal because the hand-written assembly kernels are untouched. So
the blast radius is the compiled fixed-width SIMD sorting networks specifically.
--
Reproducer:
Single file plus Highway. Highway source taken from the NumPy 2.5.1 sdist
(`numpy/_core/src/highway`) so it matches what ships, but stock Highway should do.
```cpp
// bench.cc
#include
#include
#include
#include
#include
#include "hwy/contrib/sort/vqsort.h"
#include "hwy/highway.h"
int main() {
const size_t N = 2000000;
std::mt19937_64 rng(42);
std::uniform_real_distribution d(0.0, 1.0);
std::vector master(N);
for (auto& x : master) x = d(rng);
std::vector v(N), times;
for (int r = 0; r < 7; ++r) {
v = master;
auto t0 = std::chrono::steady_clock::now();
hwy::VQSort(v.data(), N, hwy::SortAscending());
auto t1 = std::chrono::steady_clock::now();
times.push_back(std::chrono::duration(t1 - t0).count());
if (!std::is_sorted(v.begin(), v.end())) { printf("NOT SORTED\n"); return 1; }
}
std::sort(times.begin(), times.end());
printf("target=%-14s VQSort(double,2M) median %.2f ms\n",
hwy::TargetName(hwy::SupportedTargets() & ~(hwy::SupportedTargets() - 1)),
times[times.size()/2]);
return 0;
}
```
From the Highway root:
```bash
SRCS="hwy/contrib/sort/vqsort.cc hwy/contrib/sort/vqsort_f64a.cc hwy/contrib/sort/vqsort_f64d.cc \
hwy/aligned_allocator.cc hwy/targets.cc hwy/per_target.cc hwy/abort.cc hwy/contrib/sort/vqsort_have.cc"
clang++ -O3 -std=c++17 -I. -mcpu=native bench.cc $SRCS -o bench_clang && ./bench_clang
g++ -O3 -std=c++17 -I. -mcpu=native bench.cc $SRCS -o bench_gcc && ./bench_gcc
```
Environment:
clang is `clang version 24.0.0git`, LLVM monorepo trunk. GCC is `g++ (GCC) 16.1.1 20260515
(Red Hat 16.1.1-2)`. Triple `aarch64-unknown-linux-gnu`, Fedora 44, kernel 6.17.
The CPU is an NVIDIA GB10 (DGX Spark): 10x Cortex-X925 at 3.9 GHz plus 10x Cortex-A725 at 2.8 GHz,
SVE2 at 128-bit. Both compilers report `SVE2_128` as the Highway target.
Both binaries were built and run on the same physical machine, median of 7 in-process runs. GCC came
from a `fedora:44` container, clang from a container that has no GCC in it. Nothing else differed.
`VQSort` is serial here and thread count is not a factor. Neither binary links an OpenMP runtime
(nothing OpenMP- or pthread-shaped in `DT_NEEDED`), and setting `OMP_NUM_THREADS` does nothing:
```
clang OMP unset 45.39 OMP=1 45.65 OMP=4 45.63 OMP=20 45.38 ms
g++ OMP unset 29.80 OMP=1 29.69 OMP=4 29.70 OMP=20 29.52 ms
```
Time allocation:
`perf record` puts ~76% of cycles in the same function for both,
`hwy::N_SVE::detail::Recurse<..., Simd, ...>`. So both are running the intended
sorting-network code and clang's is just slower per unit of work.
Hardware counters below are from the NumPy harness rather than the standalone binary above — same two
toolchains and the same code path, but a different harness, flagging it because it matters. Identical
`np.sort` loops, 6 second windows:
```
clang 24 GCC 14.2.1
instructions 62.4 B 91.9 B
IPC 2.67 3.93
stalled-cycles-backend 51.77% 19.87%
stalled-cycles-frontend 1.41% 6.48%
branch-misses 70.4 M 309.4 M
L1-icache-load-misses 0.02% 0.01%
```
clang executes fewer instructions and misses fewer branches and is still 1.5x slower. The frontend is
fine: icache misses are negligible on both and clang's frontend stalls are lower than GCC's. The whole
delta is backend stalls, which is what you would expect from long dependency chains or high-latency
instruction selection in the sorting-network inner loops.
I have not gone further than that. Reading `Recurse` and `Sort8Rows` asm side by side is the obvious
next step and I would rather hand that to someone who knows the AArch64 scheduling model.
Ruled out:
- Target selection. Both compilers pick `SVE2_128`.
- `-mcpu` tuning. 45.31 / 45.56 ms for `-mcpu=gb10` and `-mcpu=native`. Flat.
- SVE instruction selection, apparently. `-mcpu=gb10+nosve` removes SVE from the ISA rather than
retuning, and it lands at 46.20 ms against 45.31, a 0.9 ms spread (2.0%). Highway still reports the
`SVE2_128` target either way because that is chosen by runtime detection, but the code generated
for it should differ substantially, and the timing barely moves. I would have expected `+nosve` to
either fix this or make it much worse. It does neither, which is why I am not confident the
`[AArch64][SVE2]` in the title is the right framing: the same slowdown may be present in the NEON
path too.
- Instruction cache pressure. 0.02% vs 0.01% L1i miss rate, and clang's frontend stalls are 4.6x lower.
- Thread count. `VQSort` is serial, no OpenMP runtime is linked, and `OMP_NUM_THREADS` at 1 / 4 / 20 or
unset changes nothing on either compiler (numbers above).
- NumPy's SIMD build config. `-Dcpu-dispatch=none` vs `max` is 61.19 vs 61.23 ms, `-Dcpu-baseline=min`
is 61.56 ms. No effect. (Those numbers are higher than the 47.7 ms above because they are sdist
builds with a different flag set, but they move together, which is the point.)
Notes
- The reproducer is single-file and needs only Highway, but I have not minimized below `VQSort` to a
specific function or IR pattern. Someone with the AArch64 backend in hand will get there faster.
- The counter table is from the NumPy harness, not the standalone binary. Attaching `perf` to the
short-lived standalone process returned `` and I did not chase it. All the timings are
from the standalone reproducer and are directly comparable to each other.
- clang 24.0.0git is a trunk build and I have not bisected against a released LLVM, so I cannot say
whether this is a regression or has always been the case. If a bisect helps, name the revisions and
I will run it.
- Only `double` ascending in the standalone case. At the NumPy level the gap is 1.58x for float64,
1.64x float32, 1.37x int32, 1.20x int64, so it is not one type, but I did not re-derive those
standalone.
- One machine, one core type. Not tested on Neoverse V2/N2 or any other SVE2 implementation.
Is SVE2_128 sorting-network codegen expected to trail GCC by this much? If not, where should I look.
If a bisect against released LLVM would help, lmk
Contributor guide
Research direction
Build bench.cc from the Highway root with the listed clang++ and g++ commands, then compare assembly for hwy::N_SVE::detail::Recurse and Sort8Rows. Read hwy/contrib/sort/vqsort.cc, vqsort_f64a.cc, and vqsort_f64d.cc while checking the AArch64 backend scheduling and code-generation output. Done means identifying the cause of the backend stalls and demonstrating whether a change closes the reported VQSort performance gap.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100