apache / apache/arrow

[Python] Convert array of Pandas dataframe to struct column

Open
#20,057 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Currently, converting a Pandas dataframe with a column of dataframes to Arrow fails with "Could not convert with type DataFrame: did not recognize Python value type when inferring an Arrow data type". We should be able to convert this to a List array, similar to how [the R binding do it](https://arrow.apache.org/docs/r/articles/arrow.html#r-to-arrow). This could even be bi-directional, where structs could be parsed back into a column of dataframe in `to_pandas()`

Here is an example that currently fails:

```python

import pandas as pd
import pyarrow as pa

df1 = pd.DataFrame({
'x': [1, 2, 3],
'y': ['a', 'b', 'c']
})

df = pd.DataFrame({
'df': [df1]*10
})

pa.Table.from_pandas(df)
```

Here's what the other directly might look like for the same data:

```python

sub_tab = [{'x': 1, 'y': 'a'},
{'x': 2, 'y': 'b'},
{'x': 3, 'y': 'c'}]

tab = pa.table({
'df': pa.array([sub_tab]*10)
})

print(tab.schema)
# df: list>
# child 0, item: struct
# child 0, x: int64
# child 1, y: string

tab.to_pandas()
```

**Reporter**: [Will Jones](https://issues.apache.org/jira/browse/ARROW-15247) / @wjones127

**Note**: *This issue was originally created as [ARROW-15247](https://issues.apache.org/jira/browse/ARROW-15247). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the example with pandas.DataFrame, pa.Table.from_pandas(), and tab.to_pandas(), then read the linked R binding example for the intended nested representation. Trace the existing conversion entry points and determine how a dataframe column should become List; done should include a successful conversion matching the shown schema and a decided to_pandas behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.