[X86] rsqrt estimate for 1 / sqrt(x) produces incorrect results for subnormal input
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
# Description
The transformation of 1 / sqrt(x) into an `rsqrtss`-based approximation sequence produces -Inf for _positive_ denormal inputs, where the mathematically correct result is **positive** and **finite** for "large" denormals.
I think producing Inf of the correct sign for these very small inputs might be acceptable[^1], but an Inf of the incorrect sign is not.
[^1]: There is no license that allows us to do that either I think, but turning the correct result (1e18) into +Inf probably breaks fewer programs than turning it to -Inf.
Though this is also a slight stretch since the largest finite representable float (3e38) is much bigger than the correct result.
# Reproduction
Given the following IR:
```llvm
; repro.ll
target triple = "x86_64-unknown-linux"
define float @test(float %x) {
%sqrt = call arcp float @llvm.sqrt.f32(float %x)
%rsqrt = fdiv arcp float 1.0, %sqrt
ret float %rsqrt
}
declare float @llvm.sqrt.f32(float)
```
`llc repro.ll` produces the following assembly, I included the intermediate results
```asm
test: # @test
.cfi_startproc
# %bb.0:
| xmm0 = 1e-38 (subnormal)
rsqrtss %xmm0, %xmm1 # xmm1 = 1/sqrt(xmm0) + E | xmm1 = +inf
mulss %xmm1, %xmm0 # |
mulss %xmm1, %xmm0 # xmm0 = xmm0 * xmm1^2 | xmm0 = 1e38 * +inf = +inf
addss .LCPI0_0(%rip), %xmm0 # xmm0 = xmm0 - 3.0f | xmm0 = +inf - 3.0f = +inf
mulss .LCPI0_1(%rip), %xmm1 # xmm1 = xmm1 * -0.5f | xmm1 = +inf * -0.5f = -inf
mulss %xmm1, %xmm0 # xmm0 = xmm0 * xmm1 | xmm0 = +inf * -inf = -inf
retq
```
# Additional info
`rsqrtss` treats denormals as zero. Quote from the instruction spec
> When a source value is a 0.0, an ∞ of the sign of the source value is returned. A denormal source value is treated as a 0.0 (of the same sign).
`vrsqrt14ss` (where available) respects denormals, but otherwise has the same latency and throughput as `rsqrtss`
> A denormal source value is treated as zero only if DAZ bit is set in MXCSR
GCC only seems to do this transform only with at least `-funsafe-math-optimizations -fno-math-errno -ffinite-math-only`.
LLVM targeting ARM64 also produces a reasonable (finite, not nan) result.
See an example here on [Compiler Explorer](https://godbolt.org/z/aaT3bhqjf)
# Related Issues
Issue #34342 - Similar issue for the sqrt estimate path, fixed in 2018 via [D42323](https://reviews.llvm.org/D42323). That fix did not cover the 1/sqrt → rsqrt path.
Pull #167595 - Related work about which fast-math flags should enable the rsqrt transform.
Contributor guide
Research direction
Start with the repro.ll IR and run llc for the x86_64 target to reproduce the rsqrt sequence. Trace the 1/sqrt-to-rsqrt estimate path and compare its handling of positive subnormal inputs with the related sqrt estimate fix in issue #34342. Done means the generated result no longer has the incorrect negative sign for these inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- compilers
- Domain
- compilers, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100