apache / apache/arrow

[C++][Python] PyArrow Parquet writer fails writing [null] for null structs, but Datafusion works. Need to add [null] to current working [""] for NULLs.

Open
#50,470 4 comments 0 reactions 1 assignee Claimed by @fenfeng9 View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I'm getting the following error trying to write a parquet file.

`pyarrow.lib.ArrowInvalid: Column '_Id' is declare non-nullable, but contains nulls.`

I think this is happening because there might be actual nulls in the column for the purpose of making a struct array even between the outer struct and inner struct. The image below is showing is_valid: [false] which looks odd to begin with..

The entire ShareClassType field is nullable and it contains a NULL, but the writer is complaining that a non-nullable field in belonging to non-nullable inner struct contains NULLs.

Here is the schema for this item:

Image

and the data in it:

Image

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.