apache / apache/arrow

[Python] Fixed size lists of numeric types without nulls could be converted to numpy with zero-copy

Open
#35,622 7 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

Today, FixedSizeListArrays seem to demand that `zero_copy_only` be set to `False` unnecessarily.

For example:

Create a fixed size list array: 3 rows, with fixed-size of 2:
```py
import pyarrow as pa
data = pa.FixedSizeListArray.from_arrays([1, 2, 3, 4, 5, 6], 2)
print(data)
```
```

[
[
1,
2
],
[
3,
4
],
[
5,
6
]
]
```

Calling `to_numpy()` throws an error:
```py
data.to_numpy()
```
```
ArrowInvalid: Needed to copy 1 chunks with 0 nulls, but zero_copy_only was True
```

But if I work with buffers directly, I can easily get it to work:
```py
nparray = np.frombuffer(data.buffers()[2], dtype=np.int64())
print(nparray)
```
```
array([1, 2, 3, 4, 5, 6])
```

We also know enough to even give it the right ndarray shape:
```py
nparray_shaped = np.frombuffer(data.buffers()[2], dtype=np.int64()).reshape(len(data), data.type.list_size)
print(nparray_shaped)
```
```
array([[1, 2],
[3, 4],
[5, 6]])
```

---

I propose that `FixedSizeListArray.to_numpy()` should return numpy arrays with zero copy if the FixedSizeList's type is an integer or floating point type, since those are safe to convert, and if no nulls are present.

I also propose that it reshape the output to be an ndarray which matches the FixedSizeList's shape.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start at the Python FixedSizeListArray.to_numpy() entry point and compare its current zero-copy handling with the buffers and shape described in the issue. Done means null-free fixed-size lists of integer or floating-point types can return a correctly shaped zero-copy NumPy array while unsupported or null-containing cases retain appropriate behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.