[Python] How to handle chunked arrays output in pyarrow.array(...)
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
From https://github.com/apache/arrow/pull/34289#pullrequestreview-1355094099
Currently, the `pyarrow.array(..)` constructor is meant to create Array object, but can return a ChunkedArray instead in two cases: 1) the object is too big to fit into a single array (eg offset gets too large for single StringArray), and 2) the object has a `__arrow_array__` that returns a ChunkedArray.
However, if this starts to happen more and more, that can be annoying for places in our code where we assume `pyarrow.array(..)` gives us an Array, see for example https://github.com/apache/arrow/issues/33727#issuecomment-1387323624. For this specific case, we updated `pyarrow.array(..)` to special case chunked arrays with only 1 chunk to unpack this into a normal Array, since that's an easy zero-copy conversion (done in https://github.com/apache/arrow/pull/34289).
Longer term, what do we want to do with `pyarrow.array(..)` returning chunked arrays?
For example, passing a pandas.Series to `pyarrow.array(..)` can easily give a ChunkedArray:
```python
>>> arr = pa.chunked_array([[1, 2], [3, 4]])
>>> ser = pd.Series(arr, dtype=pd.ArrowDtype(arr.type))
>>> ser
0 1
1 2
2 3
3 4
dtype: int64[pyarrow]
>>> pa.array(ser)
[
[
1,
2
],
[
3,
4
]
]
```
Some thoughts:
- To ensure you can rely more on `pa.array(..)` to actually return an Array, we could concat chunked arrays in the example above, and then users could use `pa.asarray(..)` to get either Array or ChunkedArray
- Keep `pa.array(..)` as flexible giving either Array/ChunkedArray, but add other function that is more strict, or a helper that ensures we always have an Array and concats chunks if necesssary, which could then be used internally where needed.
Contributor guide
Research direction
Start by reviewing the pyarrow.array(...) behavior described in this issue and the referenced pull request 34289, including the existing single-chunk handling. Compare the proposed pa.asarray(...) and stricter-helper directions, then define the desired return type and behavior for chunked inputs before identifying implementation and test locations. Done means the API decision is documented and its behavior is covered for both oversized inputs and __arrow_array__ results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100