apache / apache/arrow

pyarrow.RecordBatch.from_pandas fails on concatenated pandas.DataFrames: TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array

Open
#41,936 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

```python
>>> import pandas as pd
>>> pd.__version__
'2.2.2'
>>> import pyarrow as pa
>>> pa.__version__
'16.1.0'
>>> df = pd.DataFrame({"1": [1]}).astype({"1": "int32[pyarrow]"})
>>> concat_df = pd.concat([df, df])
>>> pa.RecordBatch.from_pandas(df)
pyarrow.RecordBatch
1: int32
----
1: [1]

>>> pa.RecordBatch.from_pandas(concat_df)

Traceback (most recent call last):
File "", line 1, in
File "pyarrow/table.pxi", line 3289, in pyarrow.lib.RecordBatch.from_pandas
File "pyarrow/table.pxi", line 3379, in pyarrow.lib.RecordBatch.from_arrays
TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array
```

This has been observed on my **macOS Sonoma 14.5** as well as on a server running linux.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.