[Python] to_pandas fails if a record batch's length is smaller than the length of its arrays
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
It's possible to construct a record batch whose length is smaller than its child arrays. I'm not entirely sure if this is legal (I will ask on the ML). However, if it is, then to_pandas is not working correctly in this situation:
```
# Create a struct array of length 5 with one child of length 5
foo = pa.StructArray.from_arrays([pa.array([1, 2, 3, 4, 5])], names=["x"])
# Creates a struct array of length 2 with one child of length 5
bar = foo.slice(0, 2)
# Creates a record batch of length 2 with one child of length 5
baz = pa.RecordBatch.from_struct_array(bar)
baz.to_pandas()
# Traceback (most recent call last):
# File "", line 1, in
# File "pyarrow/array.pxi", line 852, in pyarrow.lib._PandasConvertible.to_pandas
# File "pyarrow/table.pxi", line 2506, in pyarrow.lib.RecordBatch._to_pandas
# File "pyarrow/table.pxi", line 4075, in pyarrow.lib.Table._to_pandas
# File "/home/pace/dev/arrow/python/pyarrow/pandas_compat.py", line 823, in table_to_blockmanager
# return BlockManager(blocks, axes)
# ^^^^^^^^^^^^^^^^^^^^^^^^^^
# File "/home/pace/miniconda3/envs/conbench3/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 1040, in __init__
# self._verify_integrity()
# File "/home/pace/miniconda3/envs/conbench3/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 1047, in _verify_integrity
# raise construction_error(tot_items, block.shape[1:], self.axes)
# ValueError: Shape of passed values is (5, 1), indices imply (2, 1)
```
### Component(s)
Python
Contributor guide
Research direction
Start with python/pyarrow/pandas_compat.py, especially table_to_blockmanager, and reproduce the issue using the RecordBatch snippet in the report. Check how the child array shape reaches pandas when the batch length is smaller; done means baz.to_pandas() completes and returns the expected two-row result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100