llvm / llvm/llvm-project

[X86][SelectionDAG] llvm.frexp on vectors: type legalizer assertion on main, silent miscompile in 22.1, when the exponent vector needs widening

Open
#224,127 1 comment 0 reactions 0 assignees View on GitHub
backend:X86 confirmed crash-on-valid llvm:SelectionDAG
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

`llvm.frexp` on vector types asserts in the type legalizer on `main`, and silently produces wrong code in the 22.1 release, whenever the `` exponent result needs widening.

### Reproducer

```llvm
target triple = "x86_64-unknown-linux-gnu"
declare { <2 x double>, <2 x i32> } @llvm.frexp.v2f64.v2i32(<2 x double>)
define void @vfrexp2(ptr %m, ptr %e, ptr %a) {
%x = load <2 x double>, ptr %a
%r = call { <2 x double>, <2 x i32> } @llvm.frexp.v2f64.v2i32(<2 x double> %x)
%mm = extractvalue { <2 x double>, <2 x i32> } %r, 0
%ee = extractvalue { <2 x double>, <2 x i32> } %r, 1
store <2 x double> %mm, ptr %m
store <2 x i32> %ee, ptr %e
ret void
}
```

### On `main` (24.0.0git, assertions enabled)

```
llc -mcpu=x86-64 -filetype=obj -o /dev/null f2.ll

llc: llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.cpp:850:
void llvm::DAGTypeLegalizer::SetWidenedVector(SDValue, SDValue):
Assertion `Result.getValueType() == TLI.getTypeToTransformTo(*DAG.getContext(),
Op.getValueType()) && "Invalid type for widened vector"' failed.
Running pass 'X86 DAG->DAG Instruction Selection' on function '@vfrexp2'
```

### On 22.1.8 (release, assertions disabled) — silent miscompile

It compiles, and the exponent vector comes out as the bit pattern of the mantissa vector. Calling the above with `{1.0, 8.0}`:

```
frexp(1) = {0x1p-1, 0} correct {0x1p-1, 1}
frexp(8) = {0x1p-1, 1071644672} correct {0x1p-1, 4}
```

`1071644672` is `0x3FE00000`, the high half of `0.5` as an `f64`. Both mantissas are `0.5`, so `<2 x double> {0x3FE0…, 0x3FE0…}` bitcast to `<4 x i32>` is `{0, 0x3FE00000, 0, 0x3FE00000}`, and the stored `<2 x i32>` is its low half. The second result is taken from the register holding the first. The mantissa result is always correct.

### Which widths

| intrinsic | `-mcpu=x86-64` | `-mcpu=znver3` |
|---|---|---|
| `llvm.frexp.v2f64.v2i32` | assert / miscompile | assert / miscompile |
| `llvm.frexp.v4f64.v4i32` | assert / miscompile | ok |
| `llvm.frexp.v8f64.v8i32` | assert / miscompile | ok |

The pattern fits the exponent vector needing to be widened: `<2 x i32>` is 64 bits and always needs it; `<4 x i32>` and `<8 x i32>` are fine once a 256-bit register is available for the corresponding double vector. On 22.1 the crashing and the miscompiling cases are the same cases.

`llvm.modf.v2f64` and `llvm.sincos.v2f64` are both correct at every level tested. Those return two same-typed vectors, so neither result needs widening, which is consistent with the above.

### How it turned up

An OpenCL conformance failure. PoCL lowers OpenCL `frexp` to `llvm.frexp`, and its work-item loop vectorizer widens it to the work-group width; at an SSE2-only target that is `v2f64`, and every `frexp` in the kernel returned a wrong exponent while the mantissa stayed right. `frexp` became vectorizable in #112408, so the vectorizer now forms these nodes on targets whose legalizer does not handle the narrow second result.

Bisecting which commit introduced it, and the fix itself, are beyond what I can usefully do here; the reproducer above is deterministic on both versions.

*Investigated with the assistance of Claude Code; every result above was produced by running the commands shown.*

Contributor guide

Open the contributing guide

Research direction

Start with the f2.ll reproducer and the shown llc command, then inspect llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.cpp around SetWidenedVector while running the X86 DAG-to-DAG instruction-selection pass. Done means llvm.frexp.v2f64.v2i32 no longer asserts and stores the correct exponent vector on the x86-64 cases described, without regressing the working widths.

Written by the indexing model from the issue text.

Assessment

Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.