apache / apache/arrow

[C++][Python] Casting nested structs fails when inner struct are non-nullable, but the field for the inner struct is nullable

Open
#50,515 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Is this a bug and if it isn't is there a preferred way of handling this? I've got some rather complex data structures I'm trying to save into parquet using a common schema and using cast() on a struct array would be the preferred way to align data.

Shouldn't NULL in an inner array be acceptable when its validity mask is set to False? I know I can call pyarrow.compute.fill_null() and replace the NULL with a ZERO, but this can get complicated when I got thousands of fields with a mix of all kinds of datatypes: int, string, decimal, date, timestamp, boolean, etc.. and there isn't a clear way to stuff dummy scalars: pc.scalar(0), pc.scalar(""), pc.scalar(True), etc.. for every possible datatype in the Array ecosystem to replace NULL.

Here are working and non-working example using cast() on a the same struct array type.

Define struct types and setup data using a pylist vs a list of arrays. inner_struct is nullable in the outer struct definition.

Image

If we examine the two structs, to_pylist() returns the same values, but the inner struct for struct a contains a "0" while struct b has a "null".

Image

Casting both structs to the same outer struct type fails for struct b because it has a NULL value in it even though the field it belongs to in the outer struct is nullable and the NULL entry has a False validity mask.

Image

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Research direction

No source file or test is identified in the issue. Start by reproducing the Python cast examples described in the report, then trace the C++ and Python casting paths for nested structs with nullable outer fields and non-nullable inner fields. Done means the reported cast succeeds when the inner value is null but the outer validity mask is false, with regression coverage for the contrasting cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.