llvm / llvm/llvm-project

[CodeGenPrepare] Invert icmp and swap select operands when the inverse predicate lowers more cheaply

Open
#216,534 0 comments 0 reactions 0 assignees View on GitHub
llvm:codegen missed-optimization
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

LLVM can miss a target-specific optimization where an icmp predicate has a cheaper complementary form and the logical inversion can be absorbed for free by swapping the operands of a select.

For example:

```llvm
target triple = "x86_64-unknown-linux-gnu"

define <4 x i8> @src(
<8 x i8> %a,
<8 x i8> %b,
<8 x i8> %c,
<8 x i8> %d) {
entry:
%cmp = icmp ult <8 x i8> %a, %b
%cond = shufflevector <8 x i1> %cmp, <8 x i1> poison,
<4 x i32>
%tv = shufflevector <8 x i8> %c, <8 x i8> poison,
<4 x i32>
%fv = shufflevector <8 x i8> %d, <8 x i8> poison,
<4 x i32>
%r = select <4 x i1> %cond, <4 x i8> %tv, <4 x i8> %fv
ret <4 x i8> %r
}
```

This is equivalent to:

```llvm
define <4 x i8> @tgt(
<8 x i8> %a,
<8 x i8> %b,
<8 x i8> %c,
<8 x i8> %d) {
entry:
%cmp = icmp uge <8 x i8> %a, %b
%cond = shufflevector <8 x i1> %cmp, <8 x i1> poison,
<4 x i32>
%tv = shufflevector <8 x i8> %d, <8 x i8> poison,
<4 x i32>
%fv = shufflevector <8 x i8> %c, <8 x i8> poison,
<4 x i32>
%r = select <4 x i1> %cond, <4 x i8> %tv, <4 x i8> %fv
ret <4 x i8> %r
}
```

Alive2:

https://alive2.llvm.org/ce/z/WMNkpk

The general transform is:

select (icmp P A, B), X, Y
->
select (icmp inverse(P) A, B), Y, X

so the logical inversion does not require an additional IR operation.

On AVX2 this can be profitable because some unsigned packed integer comparison predicates require different synthesis sequences.

For example, with:

llc test.ll -O3 -mcpu=haswell

the ult form lowers to:

```asm
vpmaxub xmm1, xmm0, xmm1
vpcmpeqb xmm0, xmm0, xmm1
vpcmpeqd xmm1, xmm1, xmm1
...
vpxor xmm0, xmm0, xmm1
...
vpblendvb ...
```

while the equivalent uge form avoids the explicit mask inversion:

```asm
vpmaxub xmm1, xmm0, xmm1
vpcmpeqb xmm0, xmm0, xmm1
...
vpblendvb ...
```

godbolt: https://godbolt.org/z/njfz7zrbr

In the complete reproducer above, I get:

ult form: 10 instructions before ret
uge form: 8 instructions before ret

LLVM's X86 TTI also already models the predicate cost difference. Using the same fixed x86 cost model, the post-O2 forms score:

ult form: 8
uge form: 7

-O2 and -O3 currently leave the original ult/select orientation unchanged.

Why this should probably be a late target-aware transform

I don't think this belongs in InstCombine.

The preferred predicate polarity is target-dependent. On AVX2, for example, one predicate may require an extra synthesized mask inversion, while on a target with direct support for both predicates the profitability difference may disappear.

Performing this kind of rewrite in the target-independent middle end would also interfere with LLVM's canonicalization strategy. InstCombine deliberately tries to keep comparisons in stable canonical forms so that subsequent transforms do not need to handle both equivalent predicate orientations. A target-specific rewrite performed too early could therefore either:

make canonical IR target-dependent, or
simply be undone by a later canonicalization.

It seems preferable to preserve LLVM's normal canonical form throughout the middle end and only make the target-specific polarity choice near the end of IR optimization:

target-independent optimization / canonicalization

canonical icmp/select

late target-aware profitability adjustment

instruction selection

CodeGenPrepare seems like a natural place to consider this, since this is fundamentally a code-generation preparation decision rather than a target-independent algebraic canonicalization.

Conceptually, a late transform could compare the cost of the current predicate against its inverse and perform:

select (icmp P A, B), X, Y

select (icmp inverse(P) A, B), Y, X

only when the target reports a strict profitability improvement.

For the simplest implementation, this could initially be restricted to an icmp with a single select use. Shared comparisons would need additional care to avoid making other users more expensive.

I think the important part is that this should remain target-cost-driven, rather than hardcoding a preference such as UGE over ULT. The AVX2 example is just one concrete case where LLVM's existing target cost model already exposes a profitable complementary predicate.

Assisted by gpt-5.6-sol

Contributor guide

Open the contributing guide

Research direction

Start in CodeGenPrepare and inspect the existing x86 TTI predicate-cost modeling described in the issue, using the AVX2 `llc test.ll -O3 -mcpu=haswell` reproducer as a baseline. Done means a late, target-cost-driven inversion of a single-use icmp/select when its inverse is strictly cheaper, with the profitable AVX2 case covered and equal-cost cases left unchanged.

Written by the indexing model from the issue text.

Assessment

Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.