[ConstantFolding] `llvm.is.fpclass` mis-folds x87 pseudo-NaN (`x86_fp80`) as signaling
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
Constant folding of `llvm.is.fpclass` gives the wrong answer for an x87 pseudo-NaN (`x86_fp80` with the explicit integer/J-bit clear). It disagrees with the runtime lowering in `TargetLowering::expandIS_FPCLASS`, so the result changes depending on whether the operand is constant-folded.
## Reproducer
```llvm
declare i1 @llvm.is.fpclass.f80(x86_fp80, i32)
; 0xK7FFF3FFFFFFFFFFFFFFF = x87 pseudo-NaN (exp all-ones, integer/J-bit = 0, payload != 0)
define i1 @issignaling_pseudo_nan() {
%r = call i1 @llvm.is.fpclass.f80(x86_fp80 0xK7FFF3FFFFFFFFFFFFFFF, i32 1) ; fcSNan
ret i1 %r
}
```
```bash
$ opt -S -passes=instcombine repro.ll
ret i1 true ; WRONG — should be false
```
## Analysis
The backend classifies a pseudo-NaN as NaN but **neither signaling nor quiet** (`expandIS_FPCLASS` handles the F80 pseudo cases; the `fcSNan`/`fcQNan` arms implicitly require the integer bit). The folder path (`ConstantFoldScalarCall` → `IEEEFloat::isSignaling()`) only checks the quiet bit and ignores the x87 integer bit, so it reports the pseudo-NaN as signaling.
**User-visible:** `__builtin_issignaling((long double)pseudo_nan)` returns `0` at `-O0` but `1` once folded (`-O1`+/LTO).
Reproduced on current main (`6793bb48ac75`) and 23.1.0-rc1.
(Related but distinct from #63938, which is about arithmetic/`opInvalidOp` semantics for these encodings, not `is_fpclass` classification.)
### Possible fix direction
Detect the x87 pseudo-NaN inside the `is_fpclass` constant folder (using existing APFloat queries — `getSemantics()`, `isNaN()`, `bitcastToAPInt()`) and fold it as `fcNan` → true, `fcSNan` → false, `fcQNan` → false, mirroring `expandIS_FPCLASS`. This needs no change to `APFloat` itself, so `isSignaling()` and its arithmetic callers are unaffected (orthogonal to #63938).
Contributor guide
Assessment
This issue has not been assessed yet.