llvm / llvm/llvm-project

[APFloat] roundToIntegral / isInteger assert on Float8E3M4 and Float6E2M3FN

Open
#224,353 0 comments 0 reactions 0 assignees View on GitHub
llvm:support
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

`IEEEFloat::roundToIntegral` adds and subtracts the magic constant
`2^(precision-1)`. For `Float8E3M4` (precision 5, largest 15.5) and
`Float6E2M3FN` (precision 4, largest 7.5) the constant itself overflows the
format, and `assert(fs == opOK)` fires on the `convertFromAPInt` of the
constant (APFloat.cpp:2452 at 5b5247c84485).

```c++
APFloat F(APFloat::Float8E3M4(), "1.0");
F.isInteger(); // Assertion `fs == opOK' failed.
```

Every finite value of these two formats is affected. With assertions off,
the constant saturates to the largest value and the result is wrong for
values with a fraction. `Float4E2M1FN` (precision 2, constant 2, largest 6)
is fine.

An early return for formats where the constant is not representable, or a
fallback that compares the value against the truncated integer, would fix
this.

Contributor guide

Open the contributing guide

Research direction

Start in APFloat.cpp:2452 and reproduce the provided Float8E3M4 and Float6E2M3FN isInteger() example with assertions enabled. Trace roundToIntegral's magic-constant conversion, then verify that both formats avoid the assertion and return correct results for fractional finite values, while Float4E2M1FN remains unaffected.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.