apache / apache/arrow

[C++][Python] .take silently overflow on list array (when casting to large_list is needed)

Open
#26,467 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Priority: Critical Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

reproducer below
```python

import numpy as np
import pyarrow as pa
arr = pa.array([np.arange(x).astype(np.int8) for x in range(6)])
nb_repeat = 2**32 // arr.offsets.to_numpy()[-1]
indices = pa.array(np.repeat(np.arange(len(arr)), nb_repeat))
big_arr = arr.take(indices)
print(big_arr.offsets[-5:])
big_arr.validate() # hopefully this can catch it

[
-21,
-16,
-11,
-6,
-1
]
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
in
6 big_arr = arr.take(indices)
7 print(big_arr.offsets[-5:])
----> 8 big_arr.validate()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.Array.validate()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Negative offsets in list array
```

and it works fine with large_array (as expected) :

```python

import numpy as np
import pyarrow as pa
arr = pa.array([np.arange(x).astype(np.int8) for x in range(6)], type=pa.large_list(pa.int8()))
nb_repeat = 2**32 // arr.offsets.to_numpy()[-1]
indices = pa.array(np.repeat(np.arange(len(arr)), nb_repeat))
big_arr = arr.take(indices)
print(big_arr.offsets[-5:])
big_arr.validate()
[
4294967275,
4294967280,
4294967285,
4294967290,
4294967295
]
```

**Reporter**: [Artem KOZHEVNIKOV](https://issues.apache.org/jira/browse/ARROW-10494) / @artemru
#### Related issues:
- [[Python] pyarrow.concat_arrays segfaults if a resulting StringArray's capacity overflows](https://github.com/apache/arrow/issues/26180) (is related to)

**Note**: *This issue was originally created as [ARROW-10494](https://issues.apache.org/jira/browse/ARROW-10494). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the Python reproducer and inspect the list-array take path that produces negative offsets when the result exceeds 32-bit capacity. Compare it with the large_list case and use Array.validate() to confirm the completed behavior preserves valid offsets without silent overflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.