apache / apache/arrow

[Python] to_pandas fails if a record batch's length is smaller than the length of its arrays

Open
#35,452 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

It's possible to construct a record batch whose length is smaller than its child arrays. I'm not entirely sure if this is legal (I will ask on the ML). However, if it is, then to_pandas is not working correctly in this situation:

```
# Create a struct array of length 5 with one child of length 5
foo = pa.StructArray.from_arrays([pa.array([1, 2, 3, 4, 5])], names=["x"])
# Creates a struct array of length 2 with one child of length 5
bar = foo.slice(0, 2)
# Creates a record batch of length 2 with one child of length 5
baz = pa.RecordBatch.from_struct_array(bar)
baz.to_pandas()
# Traceback (most recent call last):
# File "", line 1, in
# File "pyarrow/array.pxi", line 852, in pyarrow.lib._PandasConvertible.to_pandas
# File "pyarrow/table.pxi", line 2506, in pyarrow.lib.RecordBatch._to_pandas
# File "pyarrow/table.pxi", line 4075, in pyarrow.lib.Table._to_pandas
# File "/home/pace/dev/arrow/python/pyarrow/pandas_compat.py", line 823, in table_to_blockmanager
# return BlockManager(blocks, axes)
# ^^^^^^^^^^^^^^^^^^^^^^^^^^
# File "/home/pace/miniconda3/envs/conbench3/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 1040, in __init__
# self._verify_integrity()
# File "/home/pace/miniconda3/envs/conbench3/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 1047, in _verify_integrity
# raise construction_error(tot_items, block.shape[1:], self.axes)
# ValueError: Shape of passed values is (5, 1), indices imply (2, 1)
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with python/pyarrow/pandas_compat.py, especially table_to_blockmanager, and reproduce the issue using the RecordBatch snippet in the report. Check how the child array shape reaches pandas when the batch length is smaller; done means baz.to_pandas() completes and returns the expected two-row result.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.