[BUG] FLOAT32 rounding more inaccurate than necessary
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
When rounding single-precision floats to a specified number of decimal places, the result can be slightly inaccurate due to the intermediate computations being forced into FLOAT32 as well. round.cu has rounding functors for non-fixed-point types, but all of the intermediate results are in the same type as the input rather than the highest precision type, double. This means more error is introduced during the rounding computation than is necessary.
**Steps/Code to reproduce bug**
The following code demonstrates the problem:
```c++
#include
#include
#include
#include
int main(int argc, char** argv) {
auto const input =
cudf::test::fixed_width_column_wrapper{6.121944898040965e-05f};
cudf::test::print(input);
auto const result = cudf::round(input, 10, cudf::rounding_method::HALF_UP);
cudf::test::print(*result);
return 0;
}
```
Rounding the value to the tenth decimal place should round _down_ to approximately 6.12194e-05 but instead the value is rounded _up_ to approximately 6.12195e-05 as shown in the output when running the program:
```
6.1219449e-05
6.12194999e-05
```
**Expected behavior**
FLOAT32 rounding should use FLOAT64 for intermediate results during computation to try to avoid injecting errors beyond what is necessary when dealing with floating point numbers. When I manually performed the computations on this example input value for round.cu's half_up_positive logic but using double instead of float for the intermediate values, the answer came out rounded down as expected.
It seems that the functors for floating point rounding in round.cu should _not_ be using whatever the input type is but rather `double` explicitly to avoid unnecessary additional error during the computation.
Contributor guide
Assessment
This issue has not been assessed yet.