apache / apache/arrow

[Python] Cannot mix struct and non-struct, non-null values error when saving nested types with PyArrow

Open
#30,648 5 comments 4 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

When trying to save a Pandas dataframe with a nested type (list within list, list within dict) using pyarrow engine, the following error is encountered

ArrowInvalid: ('cannot mix list and non-list, non-null values', 'Conversion failed for column A with type object')

 

Repro:
```java

import pandas as pd
x = pd.DataFrame({"A": [[24, 27, [1, 1]]]})
x.to_parquet('/tmp/a.pqt', engine="pyarrow") 
```
Doing a bit of googling, it appears that this is a known Arrow shortcoming. However, this is a commonly encountered datastructure, and 'fastparquet' handles this seamlessly. Is there a proposed timeline/plan for fixing this?

**Reporter**: [Karthik](https://issues.apache.org/jira/browse/ARROW-15142)

**Note**: *This issue was originally created as [ARROW-15142](https://issues.apache.org/jira/browse/ARROW-15142). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Pandas DataFrame example with the pyarrow engine and compare its behavior with fastparquet. Trace how the nested values in column A are converted, then add coverage for mixed list and non-list values and confirm that saving the dataframe succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.