Unnecessary 2-way unrolling produces worse code for a conditional replacement loop
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
The following loop compares the low 32 bits of each `uint64_t` element and conditionally replaces the entire element:
```cpp
#include
#include
#include
void replace(uint64_t *xs, size_t len, uint32_t before, uint64_t after) {
for (size_t i = 0; i < len; ++i) {
uint32_t low;
memcpy(&low, &xs[i], sizeof(low));
if (low == before)
xs[i] = after;
}
}
```
GCC emits a compact pointer-based loop:
```asm
test rsi, rsi
je .L1
lea rax, [rdi+rsi*8]
.L4:
cmp DWORD PTR [rdi], edx
jne .L3
mov QWORD PTR [rdi], rcx
.L3:
add rdi, 8
cmp rdi, rax
jne .L4
.L1:
ret
```
Clang/LLVM instead unrolls the loop by two. This introduces a separate `len == 1` path, an odd-element remainder path, duplicated comparisons and stores, and several additional branches:
```asm
test rsi, rsi
je .LBB0_6
cmp rsi, 1
jne .LBB0_7
...
.LBB0_8:
cmp dword ptr [rdi + 8*rax], edx
je .LBB0_9
cmp dword ptr [rdi + 8*rax + 8], edx
jne .LBB0_12
...
```
This results in substantially larger and more branch-heavy code, and I observed a large performance difference compared with GCC.
The optimized LLVM IR confirms that LLVM has explicitly transformed the loop into a 2-way-unrolled main loop plus a scalar remainder iteration.
I also tested an equivalent pointer-based LLVM IR loop:
```llvm
define void @replace(ptr %data, i64 %count, i32 %from, i64 %to) {
entry:
%empty = icmp eq i64 %count, 0
br i1 %empty, label %exit, label %preheader
preheader:
%end = getelementptr i64, ptr %data, i64 %count
br label %loop
loop:
%current = phi ptr [ %data, %preheader ], [ %next, %latch ]
%value = load i32, ptr %current, align 8
%matches = icmp eq i32 %value, %from
br i1 %matches, label %store, label %latch
store:
store i64 %to, ptr %current, align 8
br label %latch
latch:
%next = getelementptr i64, ptr %current, i64 1
%finished = icmp eq ptr %next, %end
br i1 %finished, label %exit, label %loop
exit:
ret void
}
```
LLVM lowers this IR to compact code similar to GCC’s result. This indicates that the X86 backend can generate the desired loop and that the code-quality problem is caused by the earlier unrolling decision.
It appears that the loop-unroll cost model overestimates the benefit of unrolling this conditional-store loop. LLVM should consider leaving this loop rolled, or account more accurately for the additional control flow, remainder handling, and code-size cost.
godbolt : https://compiler-explorer.com/z/KWsK83WT8
related : https://github.com/llvm/llvm-project/issues/218320
Contributor guide
Assessment
This issue has not been assessed yet.