[C++][Python] PyArrow Parquet writer fails writing [null] for null structs, but Datafusion works. Need to add [null] to current working [""] for NULLs.
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I'm getting the following error trying to write a parquet file.
`pyarrow.lib.ArrowInvalid: Column '_Id' is declare non-nullable, but contains nulls.`
I think this is happening because there might be actual nulls in the column for the purpose of making a struct array even between the outer struct and inner struct. The image below is showing is_valid: [false] which looks odd to begin with..
The entire ShareClassType field is nullable and it contains a NULL, but the writer is complaining that a non-nullable field in belonging to non-nullable inner struct contains NULLs.
Here is the schema for this item:
and the data in it:
### Component(s)
C++, Python
Contributor guide
Assessment
This issue has not been assessed yet.