[Python] Fixed size lists of numeric types without nulls could be converted to numpy with zero-copy
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the enhancement requested
Today, FixedSizeListArrays seem to demand that `zero_copy_only` be set to `False` unnecessarily.
For example:
Create a fixed size list array: 3 rows, with fixed-size of 2:
```py
import pyarrow as pa
data = pa.FixedSizeListArray.from_arrays([1, 2, 3, 4, 5, 6], 2)
print(data)
```
```
[
[
1,
2
],
[
3,
4
],
[
5,
6
]
]
```
Calling `to_numpy()` throws an error:
```py
data.to_numpy()
```
```
ArrowInvalid: Needed to copy 1 chunks with 0 nulls, but zero_copy_only was True
```
But if I work with buffers directly, I can easily get it to work:
```py
nparray = np.frombuffer(data.buffers()[2], dtype=np.int64())
print(nparray)
```
```
array([1, 2, 3, 4, 5, 6])
```
We also know enough to even give it the right ndarray shape:
```py
nparray_shaped = np.frombuffer(data.buffers()[2], dtype=np.int64()).reshape(len(data), data.type.list_size)
print(nparray_shaped)
```
```
array([[1, 2],
[3, 4],
[5, 6]])
```
---
I propose that `FixedSizeListArray.to_numpy()` should return numpy arrays with zero copy if the FixedSizeList's type is an integer or floating point type, since those are safe to convert, and if no nulls are present.
I also propose that it reshape the output to be an ndarray which matches the FixedSizeList's shape.
### Component(s)
Python
Contributor guide
Research direction
Start at the Python FixedSizeListArray.to_numpy() entry point and compare its current zero-copy handling with the buffers and shape described in the issue. Done means null-free fixed-size lists of integer or floating-point types can return a correctly shaped zero-copy NumPy array while unsupported or null-containing cases retain appropriate behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100