[LoopVectorize] Produces inefficient reduction shape for fixed-size 8-lane counter loop
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
Godbolt reproducer:
https://godbolt.org/z/h66Kd55T7
## Reduced example
```cpp
void size_128_ok(const short* __restrict__ arr,
short* __restrict__ cnt,
const short* __restrict__ th) {
for (int j = 0; j < 8; ++j) {
for (int i = 0; i < 16; ++i) {
cnt[j] += arr[i * 8 + j] <= th[j];
}
}
}
void size_512_bad(const short* __restrict__ arr,
short* __restrict__ cnt,
const short* __restrict__ th) {
for (int j = 0; j < 8; ++j) {
for (int i = 0; i < 64; ++i) {
cnt[j] += arr[i * 8 + j] <= th[j];
}
}
}
```
## Direct comparison
For `size_128_ok`, Clang/AArch64 emits compact NEON code. The generated code keeps the 8 counters in one vector register and repeatedly subtracts the vector comparison mask:
```asm
size_128_ok(short const*, short*, short const*):
ldp q1, q2, [x0]
ldr q0, [x2]
ldr q3, [x1]
cmge v1.8h, v0.8h, v1.8h
cmge v2.8h, v0.8h, v2.8h
sub v1.8h, v3.8h, v1.8h
sub v1.8h, v1.8h, v2.8h
...
ldp q3, q2, [x0, #224]
cmge v3.8h, v0.8h, v3.8h
cmge v0.8h, v0.8h, v2.8h
sub v1.8h, v1.8h, v3.8h
sub v0.8h, v1.8h, v0.8h
str q0, [x1]
ret
```
For `size_512_bad`, the same source pattern with a larger fixed trip count produces a much worse code shape. Instead of a compact loop over contiguous NEON vectors, Clang constructs many vectors lane-by-lane from scalar loads and emits a very large unrolled body:
```asm
size_512_bad(short const*, short*, short const*):
ldr h1, [x0, #128]
add x9, x0, #144
add x16, x0, #160
ldr h3, [x0]
ldr h2, [x0, #384]
add x3, x0, #400
ld1 { v1.h }[1], [x9]
ldr h4, [x0, #256]
add x15, x0, #176
ld1 { v2.h }[1], [x3]
add x3, x0, #272
ldr h5, [x0, #640]
ld1 { v4.h }[1], [x3]
...
```
The full assembly for `size_512_bad` is much larger than the `size_128_ok` output, even though the intended computation is the same 8-lane counter update pattern with a larger fixed trip count.
## Effect of changing the loop order
Changing the loop order can also trigger the bad pattern even at size 128:
```cpp
void size_128_bad(const short* __restrict__ arr,
short* __restrict__ cnt,
const short* __restrict__ th) {
for (int i = 0; i < 16; ++i) {
for (int j = 0; j < 8; ++j) {
cnt[j] += arr[i * 8 + j] <= th[j];
}
}
}
```
This also produces lane-by-lane vector construction:
```asm
size_128_bad(short const*, short*, short const*):
ldr h1, [x0]
add x8, x0, #16
add x12, x0, #32
ldr h2, [x0, #128]
add x15, x0, #144
ldr h3, [x0, #2]
ld1 { v1.h }[1], [x8]
ldr h4, [x0, #130]
add x11, x0, #18
add x14, x0, #146
ld1 { v2.h }[1], [x15]
...
```
So the issue is not simply the source-level arithmetic expression. Clang appears to choose a vectorization strategy that builds transposed vectors from scalar lane inserts, and that strategy becomes especially problematic for the larger fixed trip count.
## Expected compact shape
If loop vectorization is explicitly disabled for the outer `i` loop, Clang emits the compact NEON loop shape for the size-512 case:
```cpp
void size_512_expected_shape(const short* __restrict__ arr,
short* __restrict__ cnt,
const short* __restrict__ th) {
#pragma clang loop vectorize(disable)
for (int i = 0; i < 64; ++i) {
for (int j = 0; j < 8; ++j) {
cnt[j] += arr[i * 8 + j] <= th[j];
}
}
}
```
Generated assembly:
```asm
size_512_expected_shape(short const*, short*, short const*):
ldr q1, [x2]
ldr q0, [x1]
mov x8, xzr
.LBB3_1:
ldr q2, [x0, x8]
add x8, x8, #16
cmp x8, #1024
cmge v2.8h, v1.8h, v2.8h
sub v0.8h, v0.8h, v2.8h
b.ne .LBB3_1
str q0, [x1]
ret
```
This is the code shape I would expect for the size-512 case: one vector load from `th`, one vector load/store for `cnt`, and a loop over contiguous 8-lane vectors from `arr`.
## GCC comparison
GCC/AArch64 generates the expected compact NEON loop for all variants:
- `size_128_ok`
- `size_512_bad`
- `size_128_bad`
- `size_512_expected_shape`
The generated code has the same basic shape in each case: one vector load from `arr`, one vector compare against `th`, one vector subtract into the counter vector, and one loop branch.
For example, `size_512_bad` is compiled to:
```asm
size_512_bad(short const*, short*, short const*):
ldr q31, [x1]
ldr q30, [x2]
add x2, x0, 1024
.L6:
ldr q29, [x0], 16
cmge v29.8h, v30.8h, v29.8h
sub v31.8h, v31.8h, v29.8h
cmp x2, x0
bne .L6
str q31, [x1]
ret
```
## Summary
The main issue is that Clang/AArch64 emits compact code for the fixed-size 128-element version, but emits extremely large lane-insert-heavy code for the fixed-size 512-element version.
The bad code shape is also observable when the loop order is changed at size 128. For the size-512 case, explicitly disabling loop vectorization produces the compact NEON loop shape.
## Use case
This pattern comes from a performance-sensitive loop over interleaved multi-lane data.
The reduced example uses 8 lanes, but the underlying use case is more general: multiple independent counter reductions are packed together and processed at once. This is useful when several lanes, channels, or data streams can share the same SIMD loop structure.
For this pattern, the expected AArch64 code shape is a compact NEON loop over contiguous vector-width chunks, not lane-by-lane vector construction from scalar loads.
Sample 8-lane-wise fast median algorithm
```cpp
template
auto get_8_median_array_simd(const std::array& arr) {
std::array low{};
low.fill(0);
std::array high{};
high.fill((1 << 16) - 1);
std::array mid{};
for (std::size_t iter = 0; iter < 16; ++iter) {
std::array cnt{};
for (std::size_t lane = 0; lane < 8; ++lane) {
mid[lane] =
low[lane] + static_cast(
(static_cast(high[lane]) - low[lane]) / 2);
}
// here: this is the counter-reduction pattern from the reduced reproducer
for (std::size_t i = 0; i < N; ++i) {
for (std::size_t lane = 0; lane < 8; ++lane) {
cnt[lane] += arr[i * 8 + lane] <= mid[lane] ? 1 : 0;
}
}
for (std::size_t lane = 0; lane < 8; ++lane) {
const uint16_t take = -uint16_t(cnt[lane] > N / 2);
high[lane] = (take & mid[lane]) | (~take & high[lane]);
low[lane] =
(take & low[lane]) | (~take & uint16_t(mid[lane] + 1));
}
}
return low;
}
```
This is one concrete example of the counter-reduction pattern. The algorithm performs a binary search over the value domain and, at each iteration, counts how many values in each lane are less than or equal to the current lane-specific midpoint.
The highlighted loop is the relevant part:
```cpp
// here: this is the counter-reduction pattern from the reduced reproducer
for (std::size_t i = 0; i < N; ++i) {
for (std::size_t lane = 0; lane < 8; ++lane) {
cnt[lane] += arr[i * 8 + lane] <= mid[lane] ? 1 : 0;
}
}
```
This loop is executed repeatedly with different lane-specific thresholds, so avoiding lane-by-lane vector construction is important for this use case.
Contributor guide
Research direction
Start with the Godbolt reduced C++ reproducer and the LoopVectorize entry point, comparing the size_128_ok, size_512_bad, and size_128_bad variants on AArch64. Trace why the larger or reordered loop selects lane-insert vector construction instead of contiguous loads. Done means the reported cases produce the expected compact NEON loop shape or the behavior is covered by a focused regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100