llvm / llvm/llvm-project

[X86] rsqrt estimate for 1 / sqrt(x) produces incorrect results for subnormal input

Open
#192,432 6 comments 0 reactions 0 assignees View on GitHub
backend:X86 floating-point llvm:SelectionDAG
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

# Description

The transformation of 1 / sqrt(x) into an `rsqrtss`-based approximation sequence produces -Inf for _positive_ denormal inputs, where the mathematically correct result is **positive** and **finite** for "large" denormals.
I think producing Inf of the correct sign for these very small inputs might be acceptable[^1], but an Inf of the incorrect sign is not.

[^1]: There is no license that allows us to do that either I think, but turning the correct result (1e18) into +Inf probably breaks fewer programs than turning it to -Inf.
Though this is also a slight stretch since the largest finite representable float (3e38) is much bigger than the correct result.

# Reproduction

Given the following IR:

```llvm
; repro.ll
target triple = "x86_64-unknown-linux"

define float @test(float %x) {
%sqrt = call arcp float @llvm.sqrt.f32(float %x)
%rsqrt = fdiv arcp float 1.0, %sqrt
ret float %rsqrt
}

declare float @llvm.sqrt.f32(float)
```

`llc repro.ll` produces the following assembly, I included the intermediate results

```asm
test: # @test
.cfi_startproc
# %bb.0:
| xmm0 = 1e-38 (subnormal)
rsqrtss %xmm0, %xmm1 # xmm1 = 1/sqrt(xmm0) + E | xmm1 = +inf
mulss %xmm1, %xmm0 # |
mulss %xmm1, %xmm0 # xmm0 = xmm0 * xmm1^2 | xmm0 = 1e38 * +inf = +inf
addss .LCPI0_0(%rip), %xmm0 # xmm0 = xmm0 - 3.0f | xmm0 = +inf - 3.0f = +inf
mulss .LCPI0_1(%rip), %xmm1 # xmm1 = xmm1 * -0.5f | xmm1 = +inf * -0.5f = -inf
mulss %xmm1, %xmm0 # xmm0 = xmm0 * xmm1 | xmm0 = +inf * -inf = -inf
retq
```

# Additional info

`rsqrtss` treats denormals as zero. Quote from the instruction spec

> When a source value is a 0.0, an ∞ of the sign of the source value is returned. A denormal source value is treated as a 0.0 (of the same sign).

`vrsqrt14ss` (where available) respects denormals, but otherwise has the same latency and throughput as `rsqrtss`

> A denormal source value is treated as zero only if DAZ bit is set in MXCSR

GCC only seems to do this transform only with at least `-funsafe-math-optimizations -fno-math-errno -ffinite-math-only`.
LLVM targeting ARM64 also produces a reasonable (finite, not nan) result.
See an example here on [Compiler Explorer](https://godbolt.org/z/aaT3bhqjf)

# Related Issues

Issue #34342 - Similar issue for the sqrt estimate path, fixed in 2018 via [D42323](https://reviews.llvm.org/D42323). That fix did not cover the 1/sqrt → rsqrt path.

Pull #167595 - Related work about which fast-math flags should enable the rsqrt transform.

Contributor guide

Open the contributing guide

Research direction

Start with the repro.ll IR and run llc for the x86_64 target to reproduce the rsqrt sequence. Trace the 1/sqrt-to-rsqrt estimate path and compare its handling of positive subnormal inputs with the related sqrt estimate fix in issue #34342. Done means the generated result no longer has the incorrect negative sign for these inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
compilers
Domain
compilers, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.