llvm / llvm/llvm-project

[ProfInfo][LoopRotate] Incorrect branch probabilities after loop rotation

Open
#190,948 0 comments 0 reactions 0 assignees View on GitHub
loopoptim PGO
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

### Wrong profile information after applying loop rotation

In the following IR module, the two branches created by `loop-rotate` after splitting the loop header condition into pre-header and in-loop have incorrect branch probabilities that do not reflect the behavior of the program. From a first analysis, the root cause seems to be the heuristic used by the function [updateBranchWeights](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Transforms/Utils/LoopRotationUtils.cpp#L212).

We tested this behavior using LLVM commit [5e814e26dd72d14ac1118a647210294d38c8d01e](https://github.com/llvm/llvm-project/commit/5e814e26dd72d14ac1118a647210294d38c8d01e).

```llvm
@d = local_unnamed_addr global i32 4
@f = global i8 0
@g = global i8 0
@a = local_unnamed_addr global i32 0

; Function Attrs: inlinehint
define i32 @main() local_unnamed_addr #0 !prof !29 {
bb:
store i8 -14, ptr @f, align 1
%i = load i32, ptr @d, align 4
br label %bb1

bb1: ; preds = %bb15, %bb
%i2 = phi i32 [ 0, %bb15 ], [ %i, %bb ]
br label %bb3

bb3: ; preds = %bb12, %bb1
%i4 = phi i32 [ 1, %bb1 ], [ 0, %bb12 ]
%i5 = phi i32 [ 0, %bb1 ], [ 1, %bb12 ]
%i6 = phi i16 [ 2048, %bb1 ], [ %i13, %bb12 ]
%i7 = icmp ugt i32 %i2, %i5
br i1 %i7, label %._crit_edge.preheader, label %.loopexit, !prof !30

._crit_edge.preheader: ; preds = %bb3
br label %._crit_edge

._crit_edge: ; preds = %._crit_edge, %._crit_edge.preheader
%i8 = phi i8 [ %i10, %._crit_edge ], [ -13, %._crit_edge.preheader ]
%i9 = add i8 %i8, 7
store i8 %i9, ptr @g, align 1
%i10 = load volatile i8, ptr @g, align 1
%i11 = icmp eq i8 %i10, 57
br i1 %i11, label %bb12, label %._crit_edge, !prof !31

bb12: ; preds = %._crit_edge
%i13 = add nuw nsw i16 %i6, 1536
%i14 = icmp samesign ult i16 %i6, 8448
br i1 %i14, label %bb3, label %._crit_edge1, !prof !32, !llvm.loop !33

._crit_edge1: ; preds = %bb12
br label %bb15, !llvm.loop !33

.loopexit: ; preds = %bb3
%.lcssa = phi i32 [ %i4, %bb3 ]
br label %bb15

bb15: ; preds = %.loopexit, %._crit_edge1
%i16 = phi i32 [ 0, %._crit_edge1 ], [ %.lcssa, %.loopexit ]
%i17 = load volatile i8, ptr @f, align 1
%i18 = add i8 %i17, 1
store i8 %i18, ptr @f, align 1
%i19 = icmp slt i8 %i18, -2
br i1 %i19, label %bb1, label %bb20, !prof !36

bb20: ; preds = %bb15
%.lcssa3 = phi i32 [ %i16, %bb15 ]
store i32 %.lcssa3, ptr @a, align 4
ret i32 0
}

attributes #0 = { inlinehint }

!llvm.module.flags = !{!0}

!0 = !{i32 1, !"ProfileSummary", !1}
!1 = !{!2, !3, !4, !5, !6, !7, !8, !9, !10, !11}
!2 = !{!"ProfileFormat", !"InstrProf"}
!3 = !{!"TotalCount", i64 3700}
!4 = !{!"MaxCount", i64 2700}
!5 = !{!"MaxInternalCount", i64 600}
!6 = !{!"MaxFunctionCount", i64 2700}
!7 = !{!"NumCounts", i64 5}
!8 = !{!"NumFunctions", i64 1}
!9 = !{!"IsPartialProfile", i64 0}
!10 = !{!"PartialProfileRatio", double 0.000000e+00}
!11 = !{!"DetailedSummary", !12}
!12 = !{!13, !14, !15, !16, !17, !18, !19, !20, !21, !22, !23, !24, !25, !26, !27, !28}
!13 = !{i32 10000, i64 2700, i32 1}
!14 = !{i32 100000, i64 2700, i32 1}
!15 = !{i32 200000, i64 2700, i32 1}
!16 = !{i32 300000, i64 2700, i32 1}
!17 = !{i32 400000, i64 2700, i32 1}
!18 = !{i32 500000, i64 2700, i32 1}
!19 = !{i32 600000, i64 2700, i32 1}
!20 = !{i32 700000, i64 2700, i32 1}
!21 = !{i32 800000, i64 600, i32 2}
!22 = !{i32 900000, i64 300, i32 3}
!23 = !{i32 950000, i64 300, i32 3}
!24 = !{i32 990000, i64 50, i32 5}
!25 = !{i32 999000, i64 50, i32 5}
!26 = !{i32 999900, i64 50, i32 5}
!27 = !{i32 999990, i64 50, i32 5}
!28 = !{i32 999999, i64 50, i32 5}
!29 = !{!"function_entry_count", i64 50}
!30 = !{!"branch_weights", i32 300, i32 550}
!31 = !{!"branch_weights", i32 300, i32 2700}
!32 = !{!"branch_weights", i32 250, i32 50}
!33 = distinct !{!33, !34, !35}
!34 = !{!"llvm.loop.mustprogress"}
!35 = !{!"llvm.loop.unroll.disable"}
!36 = !{!"branch_weights", i32 550, i32 50}
```
#### Detailed Analysis

The provided IR module mainly consists of three nested loops, which are the following:
- `L1`: The outermost loop, iterates 12 times, header `bb1`;
- `L2`: The middle loop, iterates 6 times, header `bb3` (only executed in the first `L1` iteration);
- `L3`: The innermost loop, iterates 60 times, header `._crit_edge` (only executed in the first `L1` iteration as nested in L2);

`L2` header's branch is reached at each iteration of `L1`, thus each iteration of `L1` contributes to the counts of this branch.

> Below, we provide the CFG of the IR before (first image) and after (second image) the application of the pass.

Image

After the application of the `loop-rotate`, the entry check for L2 (`bb3` terminator in `module.ll`) is split into two conditions. The first (`bb1` terminator in `after.ll`) checks if the inner loop has to be executed (pre-header condition, true only in the first iteration of L1). The second (`bb4` terminator in `after.ll`) checkes if L2 must iterate (in-loop condition).

Both branches created by the `loop-rotate` pass have wrong branch probability.

Image

First, inspecting the branch instruction terminating `bb1`, we observe that the newly computed branch probabilities for it are 54.55% for the true branch and 45.45% for the false branch. This is wrong because the true branch is only taken once and this block is executed an amout of times equal to the number of iterations of L1.

Then, the branch instruction terminating `bb4` (which is executed only once after the loop rotation) has branch probabilities that are exactly the opposite of what those should be. In fact, the condition always evaluates to true but, according to BPI, the false branch has 100% of probability to be taken.

#### Reproduction
We provide the command to replicate this finding starting from the given IR module.
```bash
opt before.ll -passes='loop-rotate' -S -o after.ll
```
#### BPI analysis
Following, the output of the BPI analysis performed on the IR after the application of the pass.
```bash
opt -passes='print' after.ll
```
```
[...]
edge %bb1 -> %._crit_edge.preheader.lr.ph probability is 0x45d1745d / 0x80000000 = 54.55%
edge %bb1 -> %.loopexit probability is 0x3a2e8ba3 / 0x80000000 = 45.45%
[...]
edge %bb4 -> %._crit_edge.preheader probability is 0x00000000 / 0x80000000 = 0.00%
edge %bb4 -> %..loopexit_crit_edge probability is 0x80000000 / 0x80000000 = 100.00% [HOT edge]
[...]
```
From profiles obtained by instrumenting the IR after the application of `loop-rotate`, the accurate resulting branch probabilities are the following:
```
[...]
edge %bb1 -> %._crit_edge.preheader.lr.ph probability is 0x0aaaaaab / 0x80000000 = 8.33%
edge %bb1 -> %.loopexit probability is 0x75555555 / 0x80000000 = 91.67% [HOT edge]
[...]
edge %bb4 -> %._crit_edge.preheader probability is 0x80000000 / 0x80000000 = 100.00% [HOT edge]
edge %bb4 -> %..loopexit_crit_edge probability is 0x00000000 / 0x80000000 = 0.00%
[...]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.