llvm / llvm/llvm-project

[APFloat] toString with an explicit FormatPrecision truncates the last digit for some values

Open
#224,352 0 comments 0 reactions 0 assignees View on GitHub
llvm:support
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

`AdjustToPrecision(APInt &significand, int &exp, unsigned FormatPrecision)`
in `llvm/lib/Support/APFloat.cpp` divides the significand by a power of ten
(integer division) to reduce the digits before the decimal conversion. It
removes `floor((bits - bitsRequired) * 59 / 196)` decimal digits, which
leaves `FormatPrecision` or `FormatPrecision + 1` digits. When exactly
`FormatPrecision` digits remain, the digit-level rounding in the second
`AdjustToPrecision` has no guard digit, and the result is truncated.

```
APFloat F(8999999488.0f); // 0x50061C46
SmallString<16> S;
F.toString(S, /*FormatPrecision=*/1, /*FormatMaxPadding=*/0, false);
// S == "8.0e+09"; the correctly rounded result is "9.0e+09".
F.toString(S, 6, 0, false);
// S == "8.99999e+09"; the correctly rounded result is "9.00000e+09".
```

Over 20000 random doubles and `FormatPrecision` 1..17, 2.5% of the
strings differ from the correctly rounded (half-up) result. The natural
precision (FormatPrecision = 0) still round-trips because it has spare
digits, but `toStringShortest` (#218471) returns one digit more than the
optimum for about 0.15% of doubles, and the MLIR/IR printers fall back to
hex for constants whose 6-digit form would round-trip if rounded.

A fix is to keep one more decimal digit (decrement `tensRemovable` by one),
so the digit-level rounding always sees a guard digit. This changes
`toString` output at explicit precisions wherever it truncated, so tests
in MLIR, llubi, and clang diagnostics need to be checked.

Contributor guide

Open the contributing guide

Research direction

Start in llvm/lib/Support/APFloat.cpp at AdjustToPrecision and trace the two precision-adjustment steps, then reproduce the provided APFloat examples. Verify that explicit-precision output is correctly rounded with a guard digit, and check the mentioned MLIR, llvm, and clang diagnostic tests for expected output changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.