[Python] Convert array of Pandas dataframe to struct column
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Currently, converting a Pandas dataframe with a column of dataframes to Arrow fails with "Could not convert with type DataFrame: did not recognize Python value type when inferring an Arrow data type". We should be able to convert this to a List array, similar to how [the R binding do it](https://arrow.apache.org/docs/r/articles/arrow.html#r-to-arrow). This could even be bi-directional, where structs could be parsed back into a column of dataframe in `to_pandas()`
Here is an example that currently fails:
```python
import pandas as pd
import pyarrow as pa
df1 = pd.DataFrame({
'x': [1, 2, 3],
'y': ['a', 'b', 'c']
})
df = pd.DataFrame({
'df': [df1]*10
})
pa.Table.from_pandas(df)
```
Here's what the other directly might look like for the same data:
```python
sub_tab = [{'x': 1, 'y': 'a'},
{'x': 2, 'y': 'b'},
{'x': 3, 'y': 'c'}]
tab = pa.table({
'df': pa.array([sub_tab]*10)
})
print(tab.schema)
# df: list>
# child 0, item: struct
# child 0, x: int64
# child 1, y: string
tab.to_pandas()
```
**Reporter**: [Will Jones](https://issues.apache.org/jira/browse/ARROW-15247) / @wjones127
**Note**: *This issue was originally created as [ARROW-15247](https://issues.apache.org/jira/browse/ARROW-15247). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the example with pandas.DataFrame, pa.Table.from_pandas(), and tab.to_pandas(), then read the linked R binding example for the intended nested representation. Trace the existing conversion entry points and determine how a dataframe column should become List; done should include a successful conversion matching the shown schema and a decided to_pandas behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100