[LoopVectorize] Conditional (masked) min/max reduction is not vectorized, although conditional sum and unconditional min/max both are
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
## Summary
A min/max reduction that is guarded by a condition — i.e. the common "minimum/maximum over the elements that pass a predicate" pattern — is not vectorized by the LoopVectorizer, even though two closely related shapes are:
- the **unconditional** min/max reduction (`m = min(m, a[i])`) vectorizes, and
- the **conditional sum** reduction (`if (c) s += a[i]`) vectorizes.
Only the *conditional min/max* reduction stays scalar.
Because masked-off lanes can be filled with the min/max identity element (`INT_MAX` for min, `INT_MIN` for max, `+/-inf` for FP), this reduction is legal to vectorize — and LLVM already does exactly this for the conditional **sum** case. This looks like a recognition gap rather than a legality or cost-model limitation.
Godbolt: https://godbolt.org/z/KWErzP83W
## Test cases
`imin` is a plain scalar min; the only difference between the two loops is the
`if (a[i] > 0)` guard.
```c
#include
static inline int imin(int a, int b) { return a < b ? a : b; }
// (A) Unconditional min reduction -> VECTORIZED
int min_vec(int *restrict a, int n) {
int m = INT_MAX;
for (int i = 0; i < n; i++)
m = imin(m, a[i]);
return m;
}
// (B) Conditional (masked) min reduction -> NOT vectorized
int min_nvec(int *restrict a, int n) {
int m = INT_MAX;
for (int i = 0; i < n; i++)
if (a[i] > 0)
m = imin(m, a[i]);
return m;
}
```
Compiler: `clang` trunk. Options: `-O3 -march=x86-64-v3`.
## What LLVM produces
### (A) `min_vec` — vectorized
The loop is recognized as a min reduction and lowered to packed `vpminsd` (eight `i32` lanes per instruction) plus a horizontal `vector.reduce.smin`:
```asm
.LBB0_7:
vpminsd ymm0, ymm0, ymmword ptr [rdi + rsi] ; 8 lanes at once
vpminsd ymm1, ymm1, ymmword ptr [rdi + rsi + 32]
vpminsd ymm2, ymm2, ymmword ptr [rdi + rsi + 64]
vpminsd ymm3, ymm3, ymmword ptr [rdi + rsi + 96]
sub rsi, -128
cmp rax, rsi
jne .LBB0_7
...
vpminsd xmm0, xmm0, xmm1 ; horizontal reduce
```
IR (the relevant recurrence):
```llvm
%14 = phi <8 x i32> [ splat (i32 2147483647), %9 ], [ %26, %12 ]
%22 = load <8 x i32>, ptr %18, align 4
%26 = tail call <8 x i32> @llvm.smin.v8i32(<8 x i32> %14, <8 x i32> %22)
...
%36 = tail call i32 @llvm.vector.reduce.smin.v8i32(<8 x i32> %35)
```
### (B) `min_nvec` — NOT vectorized (only scalar-unrolled)
Adding the `if (a[i] > 0)` guard defeats recognition.
The loop is unrolled x8 but every lane is still processed one element at a time with scalar `cmp`/`cmov`; there is no `<8 x i32>` type and no `vpminsd` anywhere:
```asm
.LBB0_13:
test esi, esi
cmovg eax, r9d
mov esi, dword ptr [rdi + 4*rdx + 8] ; one element
mov r9d, eax
cmp eax, esi
jb .LBB0_15
mov r9d, esi
.LBB0_15:
test esi, esi
cmovg eax, r9d
mov esi, dword ptr [rdi + 4*rdx + 12] ; next element
...
```
IR — the recurrence is `select(cond, umin(phi, v), phi)`, all scalar `i32`:
```llvm
%23 = tail call i32 @llvm.umin.i32(i32 %18, i32 %21) ; scalar min
%24 = select i1 %22, i32 %23, i32 %18 ; guarded update
```
(`smin` is narrowed to `umin` here because the guard `a[i] > 0` lets LLVM prove the running value is in `[1, INT_MAX]` — note LLVM has enough range information, it just does not vectorize.)
The vectorizer states the reason directly:
```
remark: loop not vectorized: value that could not be identified as
reduction is used outside the loop
```
## Why is this inconsistent
The same `select(cond, , phi)` shape **is** vectorized when `` is a sum:
```c
// Conditional SUM -> VECTORIZED today
int sum_positive(int *restrict a, int n) {
int s = 0;
for (int i = 0; i < n; i++)
if (a[i] > 0)
s += a[i];
return s;
}
```
So all three of these vectorize, and only the fourth does not:
| loop | vectorized? |
|---|---|
| `m = min(m, a[i])` (unconditional min) | Yes |
| `if (c) s += a[i]` (conditional sum) | Yes |
| `if (c) m = min(m, a[i])` (conditional min) | No |
| `if (c) m = max(m, a[i])` (conditional max) | No |
The conditional min/max case is the odd one out. Filling masked-off lanes with the reduction identity (`INT_MAX`/`INT_MIN`, `+inf`/`-inf`) makes it a normal min/max reduction — the same trick LLVM already applies for conditional sum.
## Suspected root cause
The difference sits in [llvm/lib/Analysis/IVDescriptors.cpp](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Analysis/IVDescriptors.cpp#L1023).
The generic conditional-reduction matcher `isConditionalRdxPattern` only accepts an inner `FAdd`/`FSub`/`FMul`/`Add`/`Sub`/`Mul` under the guarding `select` — it never matches a min/max op. And the `Instruction::Select` dispatch in `isRecurrenceInstr` only routes the add/mul-family recurrence kinds into it:
```cpp
case Instruction::Select:
if (isSubRecurrenceKind(Kind) || Kind == RecurKind::FAdd ||
Kind == RecurKind::FMul || Kind == RecurKind::Add ||
Kind == RecurKind::Mul || Kind == RecurKind::AddChainWithSubs ||
Kind == RecurKind::FAddChainWithSubs)
return isConditionalRdxPattern(I);
```
The min/max recurrence kinds (`SMin`/`SMax`/`UMin`/`UMax`/`FMin`/`FMax`) are absent here, so a guarded `select(cond, min/max, phi)` is never handed to the conditional-reduction path.
Meanwhile `getMinMaxRecurrence` (the plain min/max matcher) walks the backedge chain expecting a pure min/max op and rejects the extra guarding `select`.
That is why the conditional **sum** is recognized while the conditional **min/max** is not.
Contributor guide
Research direction
Reproduce the min_nvec and sum_positive cases with clang at -O3 -march=x86-64-v3, then read llvm/lib/Analysis/IVDescriptors.cpp around isConditionalRdxPattern and isRecurrenceInstr. Compare the conditional select handling with getMinMaxRecurrence. Done means conditional min/max reductions are recognized and vectorized rather than only scalar-unrolled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, cpp
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100