llvm / llvm/llvm-project

[InstCombine] Formation of fmax ignores SNaN

Open
#170,486 0 comments 0 reactions 1 assignee Claimed by @wzssyqa View on GitHub
floating-point llvm:instcombine miscompilation
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

According to LangRef, `fmax SNaN, num -> QNaN`

```llvm
; Transforms/InstCombine/fcmp-select.ll

define double @test_fcmp_select_maxnum(double %x) {
%cmp1 = fcmp ogt double %x, 1.000000
%sel1 = select nnan nsz i1 %cmp1, double %x, double 1.000000
%cmp2 = fcmp olt double %sel1, 255.000000
%sel2 = select nnan nsz i1 %cmp2, double %sel1, double 255.000000
ret double %sel2
}
=>
define double @test_fcmp_select_maxnum(double %x) {
%sel1 = fmax nsz double %x, 1.000000
%sel2 = fmin nsz double %sel1, 255.000000
ret double %sel2
}
Transformation doesn't verify! (unsound)
ERROR: Value mismatch

Example:
double %x = #x7ff4000000000000 (SNaN)

Source:
i1 %cmp1 = #x0 (0)
double %sel1 = #x3ff0000000000000 (1)
i1 %cmp2 = #x1 (1)
double %sel2 = #x3ff0000000000000 (1)

Target:
double %sel1 = #x7ffc000000000000 (QNaN)
double %sel2 = #x406fe00000000000 (255)
Source value: #x3ff0000000000000 (1)
Target value: #x406fe00000000000 (255)
```

See discussion in #138303

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.