apache / apache/arrow

[C++] Allow safe cast from int64 to float64 in compute kernel for numerics that can be accurately represented above 2^53

Open
#35,563 6 comments 1 reaction 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

See discussion found in https://github.com/apache/arrow/issues/34901.

To summarize, safe casting of int64 to float64 for numerics greater than 2^53 is currently not supported in Arrow. However, according to the Double-precision floating point format[1] defined in the IEEE 754-2008 standard, there are certain integers that can be safely cast to floating point. From 2^54 to 2^55, the representable numbers are every 2^1. From 2^54 to 2^55, the representable numbers are every 2^2. From 2^55 to 2^56, the representable numbers are every 2^3, etc.

For example, integer 18014398509481984 can be safely cast between int64 and float64 according to the IEEE 754-2008 standard, but not in arrow. (However, integer 18014398509481983 CANNOT be cast safely).
```
>>> arr = pa.array([18014398509481984], type=pa.int64())
>>> arr.cast(pa.float64())
Traceback (most recent call last):
...
pyarrow.lib.ArrowInvalid: Integer value 18014398509481984 not in range: -9007199254740992 to 9007199254740992
```

[1] https://en.wikipedia.org/wiki/Double-precision_floating-point_format

### Component(s)

C++

Contributor guide

Open the contributing guide

Research direction

Start with the discussion in issue 34901 and the C++ compute kernel's current int64-to-float64 safe-cast behavior. Verify IEEE 754 representability boundaries using the examples in this issue; done means representable int64 values above 2^53 are accepted while values such as 18014398509481983 remain rejected.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.