[Python] PyArrow copies large BinaryView buffers during construction and scalar extraction
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
In SedonaDB we're using binaryview to pass around large buffers, and in the process discovered a number of places where these buffers are copied. This is a bit of a non-standard use of the BinaryView so it's no problem, but the copy during scalar extraction was surprising (it seems like this would be a useful feature to directly access the view's buffer, possibly for regular binary/string arrays as well). Our workaround is at https://github.com/apache/sedona-db/pull/999 but here's a more minimal reproducer:
```python
import pyarrow as pa
import numpy as np
big_bytes_array = b"124938ls" * 100
buf = np.arange(1000, dtype=np.uint8)
# Creating via a memoryview doesn't keep the original memory
pa_array = pa.array([memoryview(buf)], pa.binary_view())
buf_from_pa_array_via_memoryview = np.frombuffer(pa_array.buffers()[2])
np.shares_memory(buf_from_pa_array_via_memoryview, buf)
#> False
# You can force this by creating a binary array manually and casting to a view
pa_array = pa.Array.from_buffers(
type=pa.binary(),
length=1,
buffers=[
None,
pa.py_buffer(np.array([0, 1000], dtype=np.int32())),
pa.py_buffer(buf),
],
).cast(pa.binary_view())
buf_from_pa_array_via_memoryview = np.frombuffer(pa_array.buffers()[2])
np.shares_memory(buf_from_pa_array_via_memoryview, buf)
# However, the act of extracting a scalar forces a copy
buf_from_pa_array_scalar = np.frombuffer(pa_array[0].as_buffer())
np.shares_memory(buf_from_pa_array_scalar, buf)
#> False
```
### Component(s)
Python
Contributor guide
Research direction
Start by running the minimal PyArrow and NumPy reproducer, focusing on pa_array[0].as_buffer() and the resulting memory sharing. Trace the Python BinaryView scalar-extraction path and determine how the view's original buffer is handled; done means extraction can expose the existing buffer without an unexpected copy, with the reproducer confirming shared memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100