[X86][SelectionDAG] llvm.frexp on vectors: type legalizer assertion on main, silent miscompile in 22.1, when the exponent vector needs widening
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
`llvm.frexp` on vector types asserts in the type legalizer on `main`, and silently produces wrong code in the 22.1 release, whenever the `` exponent result needs widening.
### Reproducer
```llvm
target triple = "x86_64-unknown-linux-gnu"
declare { <2 x double>, <2 x i32> } @llvm.frexp.v2f64.v2i32(<2 x double>)
define void @vfrexp2(ptr %m, ptr %e, ptr %a) {
%x = load <2 x double>, ptr %a
%r = call { <2 x double>, <2 x i32> } @llvm.frexp.v2f64.v2i32(<2 x double> %x)
%mm = extractvalue { <2 x double>, <2 x i32> } %r, 0
%ee = extractvalue { <2 x double>, <2 x i32> } %r, 1
store <2 x double> %mm, ptr %m
store <2 x i32> %ee, ptr %e
ret void
}
```
### On `main` (24.0.0git, assertions enabled)
```
llc -mcpu=x86-64 -filetype=obj -o /dev/null f2.ll
llc: llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.cpp:850:
void llvm::DAGTypeLegalizer::SetWidenedVector(SDValue, SDValue):
Assertion `Result.getValueType() == TLI.getTypeToTransformTo(*DAG.getContext(),
Op.getValueType()) && "Invalid type for widened vector"' failed.
Running pass 'X86 DAG->DAG Instruction Selection' on function '@vfrexp2'
```
### On 22.1.8 (release, assertions disabled) — silent miscompile
It compiles, and the exponent vector comes out as the bit pattern of the mantissa vector. Calling the above with `{1.0, 8.0}`:
```
frexp(1) = {0x1p-1, 0} correct {0x1p-1, 1}
frexp(8) = {0x1p-1, 1071644672} correct {0x1p-1, 4}
```
`1071644672` is `0x3FE00000`, the high half of `0.5` as an `f64`. Both mantissas are `0.5`, so `<2 x double> {0x3FE0…, 0x3FE0…}` bitcast to `<4 x i32>` is `{0, 0x3FE00000, 0, 0x3FE00000}`, and the stored `<2 x i32>` is its low half. The second result is taken from the register holding the first. The mantissa result is always correct.
### Which widths
| intrinsic | `-mcpu=x86-64` | `-mcpu=znver3` |
|---|---|---|
| `llvm.frexp.v2f64.v2i32` | assert / miscompile | assert / miscompile |
| `llvm.frexp.v4f64.v4i32` | assert / miscompile | ok |
| `llvm.frexp.v8f64.v8i32` | assert / miscompile | ok |
The pattern fits the exponent vector needing to be widened: `<2 x i32>` is 64 bits and always needs it; `<4 x i32>` and `<8 x i32>` are fine once a 256-bit register is available for the corresponding double vector. On 22.1 the crashing and the miscompiling cases are the same cases.
`llvm.modf.v2f64` and `llvm.sincos.v2f64` are both correct at every level tested. Those return two same-typed vectors, so neither result needs widening, which is consistent with the above.
### How it turned up
An OpenCL conformance failure. PoCL lowers OpenCL `frexp` to `llvm.frexp`, and its work-item loop vectorizer widens it to the work-group width; at an SSE2-only target that is `v2f64`, and every `frexp` in the kernel returned a wrong exponent while the mantissa stayed right. `frexp` became vectorizable in #112408, so the vectorizer now forms these nodes on targets whose legalizer does not handle the narrow second result.
Bisecting which commit introduced it, and the fix itself, are beyond what I can usefully do here; the reproducer above is deterministic on both versions.
*Investigated with the assistance of Claude Code; every result above was produced by running the commands shown.*
Contributor guide
Research direction
Start with the f2.ll reproducer and the shown llc command, then inspect llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.cpp around SetWidenedVector while running the X86 DAG-to-DAG instruction-selection pass. Done means llvm.frexp.v2f64.v2i32 no longer asserts and stores the correct exponent vector on the x86-64 cases described, without regressing the working widths.
Written by the indexing model from the issue text.
Assessment
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100