apache / apache/arrow

[Python] pyarrow.compute.subtract_checked overflowing for some duration arrays constructed from numpy

Open
#35,088 3 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

In the example below, `arr2` and `arr3` are duration arrays with a single null element.

`arr2` is constructed from a list
`arr3` is constructed from a numpy array

Once constructed, they evaluate to being equal.

However, they exhibit different behavior once passed to `pyarrow.compute.subtract_checked`:

```
import pyarrow as pa
import pyarrow.compute as pc
import numpy as np

data1 = [86400000000]
data2 = [None]
data3 = np.array([None], dtype="timedelta64[ns]")

arr1 = pa.array(data1, type=pa.duration("ns"))
arr2 = pa.array(data2, type=pa.duration("ns"))
arr3 = pa.array(data3, type=pa.duration("ns"))

assert arr2 == arr3

pc.subtract_checked(arr1, arr2) # ok
pc.subtract_checked(arr1, arr3) # ArrowInvalid: overflow
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied Python reproducer and inspect the pyarrow.compute.subtract_checked entry point, comparing arrays built from the list and NumPy inputs. Done means the equivalent duration arrays no longer produce different results, and the subtraction no longer raises overflow for the NumPy-constructed array.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.