[C++][Python] Casting nested structs fails when inner struct are non-nullable, but the field for the inner struct is nullable
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Is this a bug and if it isn't is there a preferred way of handling this? I've got some rather complex data structures I'm trying to save into parquet using a common schema and using cast() on a struct array would be the preferred way to align data.
Shouldn't NULL in an inner array be acceptable when its validity mask is set to False? I know I can call pyarrow.compute.fill_null() and replace the NULL with a ZERO, but this can get complicated when I got thousands of fields with a mix of all kinds of datatypes: int, string, decimal, date, timestamp, boolean, etc.. and there isn't a clear way to stuff dummy scalars: pc.scalar(0), pc.scalar(""), pc.scalar(True), etc.. for every possible datatype in the Array ecosystem to replace NULL.
Here are working and non-working example using cast() on a the same struct array type.
Define struct types and setup data using a pylist vs a list of arrays. inner_struct is nullable in the outer struct definition.
If we examine the two structs, to_pylist() returns the same values, but the inner struct for struct a contains a "0" while struct b has a "null".
Casting both structs to the same outer struct type fails for struct b because it has a NULL value in it even though the field it belongs to in the outer struct is nullable and the NULL entry has a False validity mask.
### Component(s)
C++, Python
Contributor guide
Research direction
No source file or test is identified in the issue. Start by reproducing the Python cast examples described in the report, then trace the C++ and Python casting paths for nested structs with nullable outer fields and non-nullable inner fields. Done means the reported cast succeeds when the inner value is null but the outer validity mask is false, with regression coverage for the contrasting cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100