[CodeGenPrepare] Invert icmp and swap select operands when the inverse predicate lowers more cheaply
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
LLVM can miss a target-specific optimization where an icmp predicate has a cheaper complementary form and the logical inversion can be absorbed for free by swapping the operands of a select.
For example:
```llvm
target triple = "x86_64-unknown-linux-gnu"
define <4 x i8> @src(
<8 x i8> %a,
<8 x i8> %b,
<8 x i8> %c,
<8 x i8> %d) {
entry:
%cmp = icmp ult <8 x i8> %a, %b
%cond = shufflevector <8 x i1> %cmp, <8 x i1> poison,
<4 x i32>
%tv = shufflevector <8 x i8> %c, <8 x i8> poison,
<4 x i32>
%fv = shufflevector <8 x i8> %d, <8 x i8> poison,
<4 x i32>
%r = select <4 x i1> %cond, <4 x i8> %tv, <4 x i8> %fv
ret <4 x i8> %r
}
```
This is equivalent to:
```llvm
define <4 x i8> @tgt(
<8 x i8> %a,
<8 x i8> %b,
<8 x i8> %c,
<8 x i8> %d) {
entry:
%cmp = icmp uge <8 x i8> %a, %b
%cond = shufflevector <8 x i1> %cmp, <8 x i1> poison,
<4 x i32>
%tv = shufflevector <8 x i8> %d, <8 x i8> poison,
<4 x i32>
%fv = shufflevector <8 x i8> %c, <8 x i8> poison,
<4 x i32>
%r = select <4 x i1> %cond, <4 x i8> %tv, <4 x i8> %fv
ret <4 x i8> %r
}
```
Alive2:
https://alive2.llvm.org/ce/z/WMNkpk
The general transform is:
select (icmp P A, B), X, Y
->
select (icmp inverse(P) A, B), Y, X
so the logical inversion does not require an additional IR operation.
On AVX2 this can be profitable because some unsigned packed integer comparison predicates require different synthesis sequences.
For example, with:
llc test.ll -O3 -mcpu=haswell
the ult form lowers to:
```asm
vpmaxub xmm1, xmm0, xmm1
vpcmpeqb xmm0, xmm0, xmm1
vpcmpeqd xmm1, xmm1, xmm1
...
vpxor xmm0, xmm0, xmm1
...
vpblendvb ...
```
while the equivalent uge form avoids the explicit mask inversion:
```asm
vpmaxub xmm1, xmm0, xmm1
vpcmpeqb xmm0, xmm0, xmm1
...
vpblendvb ...
```
godbolt: https://godbolt.org/z/njfz7zrbr
In the complete reproducer above, I get:
ult form: 10 instructions before ret
uge form: 8 instructions before ret
LLVM's X86 TTI also already models the predicate cost difference. Using the same fixed x86 cost model, the post-O2 forms score:
ult form: 8
uge form: 7
-O2 and -O3 currently leave the original ult/select orientation unchanged.
Why this should probably be a late target-aware transform
I don't think this belongs in InstCombine.
The preferred predicate polarity is target-dependent. On AVX2, for example, one predicate may require an extra synthesized mask inversion, while on a target with direct support for both predicates the profitability difference may disappear.
Performing this kind of rewrite in the target-independent middle end would also interfere with LLVM's canonicalization strategy. InstCombine deliberately tries to keep comparisons in stable canonical forms so that subsequent transforms do not need to handle both equivalent predicate orientations. A target-specific rewrite performed too early could therefore either:
make canonical IR target-dependent, or
simply be undone by a later canonicalization.
It seems preferable to preserve LLVM's normal canonical form throughout the middle end and only make the target-specific polarity choice near the end of IR optimization:
target-independent optimization / canonicalization
↓
canonical icmp/select
↓
late target-aware profitability adjustment
↓
instruction selection
CodeGenPrepare seems like a natural place to consider this, since this is fundamentally a code-generation preparation decision rather than a target-independent algebraic canonicalization.
Conceptually, a late transform could compare the cost of the current predicate against its inverse and perform:
select (icmp P A, B), X, Y
→
select (icmp inverse(P) A, B), Y, X
only when the target reports a strict profitability improvement.
For the simplest implementation, this could initially be restricted to an icmp with a single select use. Shared comparisons would need additional care to avoid making other users more expensive.
I think the important part is that this should remain target-cost-driven, rather than hardcoding a preference such as UGE over ULT. The AVX2 example is just one concrete case where LLVM's existing target cost model already exposes a profitable complementary predicate.
Assisted by gpt-5.6-sol
Contributor guide
Research direction
Start in CodeGenPrepare and inspect the existing x86 TTI predicate-cost modeling described in the issue, using the AVX2 `llc test.ll -O3 -mcpu=haswell` reproducer as a baseline. Done means a late, target-cost-driven inversion of a single-use icmp/select when its inverse is strictly cheaper, with the profitable AVX2 case covered and equal-cost cases left unchanged.
Written by the indexing model from the issue text.
Assessment
- Domain
- compilers
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100