[C++][Python] .take silently overflow on list array (when casting to large_list is needed)
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
reproducer below
```python
import numpy as np
import pyarrow as pa
arr = pa.array([np.arange(x).astype(np.int8) for x in range(6)])
nb_repeat = 2**32 // arr.offsets.to_numpy()[-1]
indices = pa.array(np.repeat(np.arange(len(arr)), nb_repeat))
big_arr = arr.take(indices)
print(big_arr.offsets[-5:])
big_arr.validate() # hopefully this can catch it
[
-21,
-16,
-11,
-6,
-1
]
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
in
6 big_arr = arr.take(indices)
7 print(big_arr.offsets[-5:])
----> 8 big_arr.validate()
/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.Array.validate()
/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowInvalid: Negative offsets in list array
```
and it works fine with large_array (as expected) :
```python
import numpy as np
import pyarrow as pa
arr = pa.array([np.arange(x).astype(np.int8) for x in range(6)], type=pa.large_list(pa.int8()))
nb_repeat = 2**32 // arr.offsets.to_numpy()[-1]
indices = pa.array(np.repeat(np.arange(len(arr)), nb_repeat))
big_arr = arr.take(indices)
print(big_arr.offsets[-5:])
big_arr.validate()
[
4294967275,
4294967280,
4294967285,
4294967290,
4294967295
]
```
**Reporter**: [Artem KOZHEVNIKOV](https://issues.apache.org/jira/browse/ARROW-10494) / @artemru
#### Related issues:
- [[Python] pyarrow.concat_arrays segfaults if a resulting StringArray's capacity overflows](https://github.com/apache/arrow/issues/26180) (is related to)
**Note**: *This issue was originally created as [ARROW-10494](https://issues.apache.org/jira/browse/ARROW-10494). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the Python reproducer and inspect the list-array take path that produces negative offsets when the result exceeds 32-bit capacity. Compare it with the large_list case and use Array.validate() to confirm the completed behavior preserves valid offsets without silent overflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100