[ProfInfo][LoopRotate] Incorrect branch probabilities after loop rotation
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
### Wrong profile information after applying loop rotation
In the following IR module, the two branches created by `loop-rotate` after splitting the loop header condition into pre-header and in-loop have incorrect branch probabilities that do not reflect the behavior of the program. From a first analysis, the root cause seems to be the heuristic used by the function [updateBranchWeights](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Transforms/Utils/LoopRotationUtils.cpp#L212).
We tested this behavior using LLVM commit [5e814e26dd72d14ac1118a647210294d38c8d01e](https://github.com/llvm/llvm-project/commit/5e814e26dd72d14ac1118a647210294d38c8d01e).
```llvm
@d = local_unnamed_addr global i32 4
@f = global i8 0
@g = global i8 0
@a = local_unnamed_addr global i32 0
; Function Attrs: inlinehint
define i32 @main() local_unnamed_addr #0 !prof !29 {
bb:
store i8 -14, ptr @f, align 1
%i = load i32, ptr @d, align 4
br label %bb1
bb1: ; preds = %bb15, %bb
%i2 = phi i32 [ 0, %bb15 ], [ %i, %bb ]
br label %bb3
bb3: ; preds = %bb12, %bb1
%i4 = phi i32 [ 1, %bb1 ], [ 0, %bb12 ]
%i5 = phi i32 [ 0, %bb1 ], [ 1, %bb12 ]
%i6 = phi i16 [ 2048, %bb1 ], [ %i13, %bb12 ]
%i7 = icmp ugt i32 %i2, %i5
br i1 %i7, label %._crit_edge.preheader, label %.loopexit, !prof !30
._crit_edge.preheader: ; preds = %bb3
br label %._crit_edge
._crit_edge: ; preds = %._crit_edge, %._crit_edge.preheader
%i8 = phi i8 [ %i10, %._crit_edge ], [ -13, %._crit_edge.preheader ]
%i9 = add i8 %i8, 7
store i8 %i9, ptr @g, align 1
%i10 = load volatile i8, ptr @g, align 1
%i11 = icmp eq i8 %i10, 57
br i1 %i11, label %bb12, label %._crit_edge, !prof !31
bb12: ; preds = %._crit_edge
%i13 = add nuw nsw i16 %i6, 1536
%i14 = icmp samesign ult i16 %i6, 8448
br i1 %i14, label %bb3, label %._crit_edge1, !prof !32, !llvm.loop !33
._crit_edge1: ; preds = %bb12
br label %bb15, !llvm.loop !33
.loopexit: ; preds = %bb3
%.lcssa = phi i32 [ %i4, %bb3 ]
br label %bb15
bb15: ; preds = %.loopexit, %._crit_edge1
%i16 = phi i32 [ 0, %._crit_edge1 ], [ %.lcssa, %.loopexit ]
%i17 = load volatile i8, ptr @f, align 1
%i18 = add i8 %i17, 1
store i8 %i18, ptr @f, align 1
%i19 = icmp slt i8 %i18, -2
br i1 %i19, label %bb1, label %bb20, !prof !36
bb20: ; preds = %bb15
%.lcssa3 = phi i32 [ %i16, %bb15 ]
store i32 %.lcssa3, ptr @a, align 4
ret i32 0
}
attributes #0 = { inlinehint }
!llvm.module.flags = !{!0}
!0 = !{i32 1, !"ProfileSummary", !1}
!1 = !{!2, !3, !4, !5, !6, !7, !8, !9, !10, !11}
!2 = !{!"ProfileFormat", !"InstrProf"}
!3 = !{!"TotalCount", i64 3700}
!4 = !{!"MaxCount", i64 2700}
!5 = !{!"MaxInternalCount", i64 600}
!6 = !{!"MaxFunctionCount", i64 2700}
!7 = !{!"NumCounts", i64 5}
!8 = !{!"NumFunctions", i64 1}
!9 = !{!"IsPartialProfile", i64 0}
!10 = !{!"PartialProfileRatio", double 0.000000e+00}
!11 = !{!"DetailedSummary", !12}
!12 = !{!13, !14, !15, !16, !17, !18, !19, !20, !21, !22, !23, !24, !25, !26, !27, !28}
!13 = !{i32 10000, i64 2700, i32 1}
!14 = !{i32 100000, i64 2700, i32 1}
!15 = !{i32 200000, i64 2700, i32 1}
!16 = !{i32 300000, i64 2700, i32 1}
!17 = !{i32 400000, i64 2700, i32 1}
!18 = !{i32 500000, i64 2700, i32 1}
!19 = !{i32 600000, i64 2700, i32 1}
!20 = !{i32 700000, i64 2700, i32 1}
!21 = !{i32 800000, i64 600, i32 2}
!22 = !{i32 900000, i64 300, i32 3}
!23 = !{i32 950000, i64 300, i32 3}
!24 = !{i32 990000, i64 50, i32 5}
!25 = !{i32 999000, i64 50, i32 5}
!26 = !{i32 999900, i64 50, i32 5}
!27 = !{i32 999990, i64 50, i32 5}
!28 = !{i32 999999, i64 50, i32 5}
!29 = !{!"function_entry_count", i64 50}
!30 = !{!"branch_weights", i32 300, i32 550}
!31 = !{!"branch_weights", i32 300, i32 2700}
!32 = !{!"branch_weights", i32 250, i32 50}
!33 = distinct !{!33, !34, !35}
!34 = !{!"llvm.loop.mustprogress"}
!35 = !{!"llvm.loop.unroll.disable"}
!36 = !{!"branch_weights", i32 550, i32 50}
```
#### Detailed Analysis
The provided IR module mainly consists of three nested loops, which are the following:
- `L1`: The outermost loop, iterates 12 times, header `bb1`;
- `L2`: The middle loop, iterates 6 times, header `bb3` (only executed in the first `L1` iteration);
- `L3`: The innermost loop, iterates 60 times, header `._crit_edge` (only executed in the first `L1` iteration as nested in L2);
`L2` header's branch is reached at each iteration of `L1`, thus each iteration of `L1` contributes to the counts of this branch.
> Below, we provide the CFG of the IR before (first image) and after (second image) the application of the pass.
After the application of the `loop-rotate`, the entry check for L2 (`bb3` terminator in `module.ll`) is split into two conditions. The first (`bb1` terminator in `after.ll`) checks if the inner loop has to be executed (pre-header condition, true only in the first iteration of L1). The second (`bb4` terminator in `after.ll`) checkes if L2 must iterate (in-loop condition).
Both branches created by the `loop-rotate` pass have wrong branch probability.
First, inspecting the branch instruction terminating `bb1`, we observe that the newly computed branch probabilities for it are 54.55% for the true branch and 45.45% for the false branch. This is wrong because the true branch is only taken once and this block is executed an amout of times equal to the number of iterations of L1.
Then, the branch instruction terminating `bb4` (which is executed only once after the loop rotation) has branch probabilities that are exactly the opposite of what those should be. In fact, the condition always evaluates to true but, according to BPI, the false branch has 100% of probability to be taken.
#### Reproduction
We provide the command to replicate this finding starting from the given IR module.
```bash
opt before.ll -passes='loop-rotate' -S -o after.ll
```
#### BPI analysis
Following, the output of the BPI analysis performed on the IR after the application of the pass.
```bash
opt -passes='print' after.ll
```
```
[...]
edge %bb1 -> %._crit_edge.preheader.lr.ph probability is 0x45d1745d / 0x80000000 = 54.55%
edge %bb1 -> %.loopexit probability is 0x3a2e8ba3 / 0x80000000 = 45.45%
[...]
edge %bb4 -> %._crit_edge.preheader probability is 0x00000000 / 0x80000000 = 0.00%
edge %bb4 -> %..loopexit_crit_edge probability is 0x80000000 / 0x80000000 = 100.00% [HOT edge]
[...]
```
From profiles obtained by instrumenting the IR after the application of `loop-rotate`, the accurate resulting branch probabilities are the following:
```
[...]
edge %bb1 -> %._crit_edge.preheader.lr.ph probability is 0x0aaaaaab / 0x80000000 = 8.33%
edge %bb1 -> %.loopexit probability is 0x75555555 / 0x80000000 = 91.67% [HOT edge]
[...]
edge %bb4 -> %._crit_edge.preheader probability is 0x80000000 / 0x80000000 = 100.00% [HOT edge]
edge %bb4 -> %..loopexit_crit_edge probability is 0x00000000 / 0x80000000 = 0.00%
[...]
```
Contributor guide
Assessment
This issue has not been assessed yet.