[Python] Cannot mix struct and non-struct, non-null values error when saving nested types with PyArrow
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
When trying to save a Pandas dataframe with a nested type (list within list, list within dict) using pyarrow engine, the following error is encountered
ArrowInvalid: ('cannot mix list and non-list, non-null values', 'Conversion failed for column A with type object')
Repro:
```java
import pandas as pd
x = pd.DataFrame({"A": [[24, 27, [1, 1]]]})
x.to_parquet('/tmp/a.pqt', engine="pyarrow")
```
Doing a bit of googling, it appears that this is a known Arrow shortcoming. However, this is a commonly encountered datastructure, and 'fastparquet' handles this seamlessly. Is there a proposed timeline/plan for fixing this?
**Reporter**: [Karthik](https://issues.apache.org/jira/browse/ARROW-15142)
**Note**: *This issue was originally created as [ARROW-15142](https://issues.apache.org/jira/browse/ARROW-15142). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the Pandas DataFrame example with the pyarrow engine and compare its behavior with fastparquet. Trace how the nested values in column A are converted, then add coverage for mixed list and non-list values and confirm that saving the dataframe succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100