llvm / llvm/llvm-project

[AArch64] Loop Vectorizer fails to form interleaved store group (st2) when pointers are outer-loop inductions

Open
#211,229 12 comments 0 reactions 0 assignees View on GitHub
missed-optimization vectorizers
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

## Summary

The Loop Vectorizer fails to form a factor-2 interleaved store group when two stride-2 store pointers are carried across an outer loop. The cost model then sees the stores as expensive scalar scatters and rejects vectorization entirely as "not beneficial."

The same logical store pattern in a single (non-nested) loop correctly forms the interleave group, vectorizes, and generates `st2`.

## Reproduction

```cpp
// WORKS: vectorized with st2
void single_loop(const float* __restrict src, float* __restrict dst, unsigned n) {
float* pA = dst;
float* pB = dst + 1;
for (unsigned i = 0; i < n; i++) {
*pA = src[i] * 2.0f;
*pB = src[i] * 3.0f;
pA += 2;
pB += 2;
}
}

// FAILS: not vectorized ("not beneficial")
void nested_loop(const float* __restrict src, float* __restrict dst,
unsigned width, unsigned height, unsigned srcStride) {
float* pA = dst;
float* pB = dst + 1;
for (unsigned y = 0; y < height; y++) {
const float* row = src + y * srcStride;
for (unsigned x = 0; x < width; x++) {
*pA = row[x] * 2.0f;
*pB = row[x] * 3.0f;
pA += 2;
pB += 2;
}
}
}
```

Compile with:
```
clang --target=aarch64-linux-gnu -O2 -S test.cpp -Rpass=loop-vectorize -Rpass-missed=loop-vectorize
```

## Observed Behavior

`single_loop`: vectorized (width 4, interleaved count 2), generates optimal `st2`:
```asm
st2 { v5.4s, v6.4s }, [x11]
st2 { v3.4s, v4.4s }, [x14]
```

`nested_loop`: **not vectorized at all**:
```
remark: the cost-model indicates that vectorization is not beneficial
remark: the cost-model indicates that interleaving is not beneficial
```

If vectorization is forced with `#pragma clang loop vectorize(enable) vectorize_width(4)`, the computation is vectorized but stores are emitted as scalar lane extracts:
```asm
st1 { v3.s }[1], [x7]
st1 { v3.s }[3], [x7]
st1 { v3.s }[2], [x1]
st1 { v2.s }[1], [x7]
st1 { v2.s }[2], [x5]
st1 { v2.s }[3], [x7]
```

## Expected Behavior

`nested_loop` should vectorize and generate `st2` instructions, since the inner loop store pattern is identical to `single_loop` (two pointers offset by 4 bytes, both advancing by stride 8 bytes per iteration).

## Impact

This pattern is common in image processing code that computes two (or more) output channels stored interleaved in memory, with a 2D loop nest (rows x columns).

The performance impact is significant: `st2` writes 32 bytes in one instruction vs 6+ `st1` lane stores for the same data.

Contributor guide

Open the contributing guide

Research direction

Start with the test.cpp reproduction and run the provided AArch64 clang command, comparing vectorizer remarks and generated assembly for the single and nested loops. Trace the loop-vectorizer handling of the outer-loop induction pointers; done means nested_loop vectorizes without forcing and produces st2 stores rather than scalar lane stores.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.