LLVM performs multi-instruction MOVi64imm rematerialization inside an inner loop on AArch64
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
**Godbolt:** https://godbolt.org/z/qTqKehr4s
On AArch64, LLVM rematerializes several 64-bit constants inside an inner `do` loop.
In the generated assembly, `.LBB0_14` corresponds to the `do` loop body. Several constants are reconstructed inside this block using four-instruction sequences of the form:
```asm
mov
movk
movk
movk
```
The block ends with the loop backedge:
```asm
b.hi .LBB0_14
```
so these constant materializations execute on every iteration of the inner loop.
At least some of these constants could instead be rematerialized once in the `do` loop preheader and then kept in registers across the loop.
GCC does not rematerialize these constants inside the inner loop for the same source. It materializes the constants before the loop and keeps them in registers across the loop. The Godbolt link above contains both the Clang and GCC output for comparison.
The LLVM behavior is also sensitive to the branch probability introduced by `__builtin_expect`. If I change:
```cpp
if (LIKELY(len <= 16))
```
to:
```cpp
if (len <= 16)
```
LLVM keeps/materializes the relevant constants outside the inner loop instead.
Similarly, if I change the branch weights to 90:10, LLVM keeps/materializes the relevant constants outside the inner loop instead.
* **LLVM without `LIKELY`:** https://godbolt.org/z/zsxv5rhrY
* **LLVM with 90:10 branch weights:** https://godbolt.org/z/8WT3j8aM5
I originally observed this behavior in Folly's rapidhash implementation:
https://github.com/facebook/folly/blob/main/folly/external/rapidhash/rapidhash.h#L167
The testcase in the Godbolt link is a slightly reduced version that preserves the behavior. Further reductions to the cold path changed the register pressure and no longer reproduced the inner-loop rematerialization.
## Initial investigation
The issue appears to involve the spill/splitting cost model for rematerializable constants under the very low block frequency of the unlikely path.
With `LIKELY(len <= 16)`, LLVM assigns branch weights of 2000:1 (~99.95% / 0.05%), so the path containing the inner loop has a very small entry-relative block frequency. This substantially reduces the spill weights of intervals whose uses are on that path.
The 2000:1 likely/unlikely ratio also seems fairly aggressive here and may be contributing to the interaction. For comparison, GCC currently uses a default probability of 90% for `__builtin_expect`. The rationale for GCC's value was based on weighted FDO counter data from several workloads:
https://github.com/gcc-mirror/gcc/commit/942df7390fb64e26bb7ed6ac58ee9911be4ea5cb#diff-f66b4cc2e358ae9ef4505201c7bf1b8cdec3f5b8fc1c94cb7cda21c7a28a0fb1R402
There are two additional details that seem relevant:
* `CalcSpillWeights` multiplies the weight of a fully rematerializable live interval by `0.5`, making it a preferred candidate for spilling. There is already a FIXME around this for non-trivial rematerialization.
* During region splitting, `calcBlockSplitCost()` assumes that isolating a block normally requires one spill instruction (a load or store), and charges one block-frequency unit. For these AArch64 constants, however, the `MOVi64imm` pseudo expands to four real instructions (`mov` + three `movk`s).
Looking at one of the affected constants, Greedy initially creates a live interval spanning the inner loop and successfully allocates it to a register. It is later evicted by another interval, after which Greedy splits it between its uses inside the loop. It then rematerializes the original `MOVi64imm` at the new interval boundary inside the loop.
A preferable outcome would be to rematerialize the constant once in the `do` loop preheader and keep that value live across the loop, rather than reconstructing it on every iteration.
I'm not sure what the appropriate fix in the cost model here would be
Contributor guide
Research direction
Start with the reduced C++ testcase and the Clang AArch64 output in the linked Godbolt examples, comparing the default, no-LIKELY, and 90:10 branch-weight cases. Investigate the spill/splitting cost model, including CalcSpillWeights and calcBlockSplitCost(), and the MOVi64imm rematerialization behavior. Done means the affected constants are not reconstructed inside the inner loop while the relevant compiler tests and assembly behavior remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100