llvm / llvm/llvm-project

LoopVectorize: conditional loop-invariant load is scalarize-and-predicated, blocks vectorization

Open
#200,355 0 comments 0 reactions 0 assignees View on GitHub
vectorizers
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

When a loop contains a loop-invariant load whose execution is guarded by a loop-varying condition, the loop vectorizer models that load as a per-lane scalarize-and-predicate operation (REPLICATE with S->V) and assigns it a very high cost.

That single cost dominates the cost model, so such loops stay scalar at -O3. Even when vectorization is forced, the vector body emits one predicated code which is strictly worse than the scalar loop.

This pattern (conditionally using a loop-invariant value) is very common, so the cost model is both blocking auto-vectorization and producing bad codegen when users opt in.

```c
void foo(int *A, int *B, int *C, int *D, int *E, unsigned len) {
for (unsigned i = 0; i < len; i++)
if (A[i])
B[i] = C[i] + D[0];
}
```

Command:
`clang test.c -O3 -c -S -emit-llvm -o out.ll -mavx2 -mllvm -debug-only=loop-vectorize`

Problem
The conditional load of the loop-invariant address D[0] (under if (A[i])) is modeled as scalarize-and-predicate, which gets a huge cost and blocks vectorization at -O3:

```
Cost of 3000000 for VF 2: REPLICATE ir<%2> = load ir<%D> (S->V) ...
Cost of 3000000 for VF 4: REPLICATE ir<%2> = load ir<%D> (S->V) ...
Cost of 3000000 for VF 8: REPLICATE ir<%2> = load ir<%D> (S->V) ...
LV: Selecting VF: 1.
LV: Vectorization is possible but not beneficial.

Forcing vectorization with -mllvm -force-vector-width=4 confirms it: the vector body emits one predicated scalar load of D per lane (4 loads + 4 extract/branch/insert per iteration) instead of a single hoisted load + broadcast:
pred.load.if: %5 = load i32, ptr %D, align 4
pred.load.if23: %9 = load i32, ptr %D, align 4
pred.load.if25: %13 = load i32, ptr %D, align 4
pred.load.if27: %17 = load i32, ptr %D, align 4
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.