llvm / llvm/llvm-project

[InstCombine] Canonicalize nsz NaN-fallback [0,1] float clamp to minnum/maxnum

Open
#213,189 1 comment 0 reactions 1 assignee Claimed by @junyeong0619 View on GitHub
llvm:instcombine
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

## Summary

LLVM does not canonicalize a hand-written `nsz` "clamp a non-NaN input to `[0, 1]`,
and return a fallback when the input is NaN" idiom — built entirely out of
`fcmp`/`select` — into the standard `llvm.minnum`/`llvm.maxnum` form plus a single
explicit NaN check. Under `nsz` (no-signed-zeros), the two forms are value-equivalent.

## Reduced POC

Current form (scalar, `LO = 0.0`, `HI = 1.0`):

```llvm
declare float @llvm.minnum.f32(float, float)
declare float @llvm.maxnum.f32(float, float)

define float @current_scalar(float %x, float %fallback) {
%below_lo_or_nan = fcmp nsz ult float %x, 0.0
%below_hi = fcmp nsz olt float %x, 1.0
%upper = select nsz i1 %below_hi, float %x, float 1.0
%ordered = fcmp ord float %x, 0.0
%lower_or_fallback = select nsz i1 %ordered, float 0.0, float %fallback
%result = select nsz i1 %below_lo_or_nan, float %lower_or_fallback, float %upper
ret float %result
}
```

The real-world instances (see Corpus evidence) additionally carry the full
`-ffast-math` flag set (`reassoc nsz arcp contract afn`); the transform itself only
relies on `nsz` on the value-producing `select`s.

## Proposed / Expected Result

```llvm
define float @proposed_scalar(float %x, float %fallback) {
%isnan = fcmp uno float %x, %x
%upper = call nsz float @llvm.minnum.f32(float %x, float 1.0)
%clamped = call nsz float @llvm.maxnum.f32(float %upper, float 0.0)
%result = select i1 %isnan, float %fallback, float %clamped
ret float %result
}
```

`opt -O2` on current main does **not** perform this fold; the `fcmp`/`select` tree is
left untouched. A related `nnan`+`nsz` select-to-`minnum`/`maxnum` canonicalization
already exists in `InstCombineSelect.cpp`; the missed case is the explicit
**NaN-fallback** form, which handles NaN with a `select` instead of requiring `nnan`.

## Why the relation holds

Evaluate the source tree case by case (`LO = 0.0`, `HI = 1.0`):

| input `x` | source result | reason |
|----------------|---------------|---------------------------------------------------------------|
| `NaN` | `fallback` | `ult` is unordered → true; `ord` false → picks `fallback` |
| `x < LO` | `LO` | `below_lo_or_nan` true, not NaN → picks `LO` |
| `LO <= x < HI` | `x` | `below_lo_or_nan` false → `upper`, and `below_hi` true → `x` |
| `x >= HI` | `HI` | `below_lo_or_nan` false → `upper`, and `below_hi` false → `HI`|

That is exactly `isnan(x) ? fallback : clamp(x, LO, HI)`, which is what the target form
computes: `fcmp uno x, x` selects `fallback` on NaN, otherwise
`maxnum(minnum(x, HI), LO)` clamps to `[LO, HI]`.

The two forms can differ only in the **sign of a zero** result (`minnum`/`maxnum` of
`+0.0`/`-0.0`), which is exactly what the `nsz` flag on every value-producing `select`
allows us to ignore. `nsz` on the `fcmp`s is irrelevant.

Alive2 proof (minimal `nsz`-only form — no other fast-math flags needed):
https://alive2.llvm.org/ce/z/NL6Xsh

## Backend and cost evidence

Measured with `llvm-mca --iterations=100` (so instruction/uOp counts are for 100
iterations), on the scalar forms above and their `<8 x float>` (`v8`) vector analogues:

| Target | Variant | Form | Instructions | Total Cycles | Total uOps | Block RThroughput |
|--------|---------|------|--------------|--------------|------------|-------------------|
| x86-64-v4 | scalar | current | 800 | 1004 | 1100 | 4.0 |
| x86-64-v4 | scalar | proposed | 700 | 1404 | 1000 | 2.0 |
| x86-64-v4 | v8 | current | 800 | 1204 | 1100 | 2.0 |
| x86-64-v4 | v8 | proposed | 600 | 1504 | 900 | 1.5 |
| AArch64 Neoverse V1 | scalar | current | 800 | 603 | 800 | 2.0 |
| AArch64 Neoverse V1 | scalar | proposed | 700 | 605 | 700 | 1.0 |
| AArch64 Neoverse V1 | v8 | current | 1200 | 604 | 1200 | 2.8 |
| AArch64 Neoverse V1 | v8 | proposed | 1100 | 605 | 1100 | 2.5 |
| RV64 P670 (RVV) | scalar | current | 1400 | 358 | 1400 | 3.5 |
| RV64 P670 (RVV) | scalar | proposed | 900 | 606 | 900 | 2.3 |
| RV64 P670 (RVV) | v8 | current | 1100 | 804 | 1100 | 5.0 |
| RV64 P670 (RVV) | v8 | proposed | 1000 | 606 | 1000 | 3.0 |

The proposed form reduces **instructions, uOps, and block reciprocal throughput in
every measured comparison**. Cycle counts are mixed across targets (x86-64-v4 and RV64
scalar increase, AArch64 is roughly flat, RV64 vector decreases). At the instruction
level this reflects a branchless win on AArch64 (`fcmp; fminnm; fmaxnm; fcsel`), while on
x86-64 the explicit NaN check can lower to a branch — see the open question below.

## Corpus evidence

This pattern was identified by corpus scanning (@ParkHanbum): 8 exact instances of
this tree appear in darktable's optimized IR
(`bench/darktable/optimized/introspection_dither.ll`), in image-processing regions.
These instances carry the full `-ffast-math` flag set, consistent with the `nsz`
precondition of the transform.

## Suggested scope

* Guard on `nsz` on every value-producing `select`.
* Guard on `denormal-fp-math = ieee`: `-ffast-math` (the usual source of `nsz`)
commonly links `crtfastmath.o`, enabling FTZ/DAZ, under which flushing denormals
could make the `minnum`/`maxnum` form observably differ from the original.
* Literal bounds `0.0`/`1.0` in this POC, but the relation holds for any `LO <= HI`.
* Scalar or fixed-vector `float` only; excludes strict/constrained FP, scalable
vectors, and `minimum`/`maximum`.
* Negative cases that each break one precondition (missing `nsz`, wrong predicate,
mismatched `LO` constant, mismatched compared operand, non-`ieee` denormal mode)
pin down the match exactly.

## Open question

Given the target-dependent cost (a branchless win on AArch64, but an added NaN branch
on x86-64), should this be an unconditional InstCombine canonicalization, or should it
be target-guarded?

cc @ParkHanbum

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.