[Python] pyarrow.compute.subtract_checked overflowing for some duration arrays constructed from numpy
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
In the example below, `arr2` and `arr3` are duration arrays with a single null element.
`arr2` is constructed from a list
`arr3` is constructed from a numpy array
Once constructed, they evaluate to being equal.
However, they exhibit different behavior once passed to `pyarrow.compute.subtract_checked`:
```
import pyarrow as pa
import pyarrow.compute as pc
import numpy as np
data1 = [86400000000]
data2 = [None]
data3 = np.array([None], dtype="timedelta64[ns]")
arr1 = pa.array(data1, type=pa.duration("ns"))
arr2 = pa.array(data2, type=pa.duration("ns"))
arr3 = pa.array(data3, type=pa.duration("ns"))
assert arr2 == arr3
pc.subtract_checked(arr1, arr2) # ok
pc.subtract_checked(arr1, arr3) # ArrowInvalid: overflow
```
### Component(s)
Python
Contributor guide
Research direction
Start by running the supplied Python reproducer and inspect the pyarrow.compute.subtract_checked entry point, comparing arrays built from the list and NumPy inputs. Done means the equivalent duration arrays no longer produce different results, and the subtraction no longer raises overflow for the NumPy-constructed array.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100