apache / apache/arrow-rs

`StructArray` allow to build with child that have nulls but for non nullable field

Open
#9,302 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**I opened a mailing list discussion on what the behavior should be**
[here](https://lists.apache.org/thread/7gbqjwykh1ob3xbvwph3ljsdl5c7kxpd)

**Describe the bug**
You can create a `StructArray` with field marked as non nullable but the child has nulls.

this should not be possible IMO as:
1. struct field nullability has no meaning if the array can have nulls (I'm aware that the array can only have nulls in the same positions as the struct nulls)
2. you can't have functions asserting non nullable field array has no nulls
3. You can't work on the child arrays in a separate context without the data of the parent

**To Reproduce**
```rust
#[test]
fn reproduction() {
let child = Int8Array::from(vec![None]);
let child = Arc::new(child) as ArrayRef;
let struct_array = StructArray::try_new(
Fields::from(vec![
Field::new(
"f",
child.data_type().clone(),
// Note the false here
false
),
]),
vec![child],
// Creating null item
Some(NullBuffer::from(&[false])),
).expect_err("should fail as field nullability does not match the array nullability");
}
```

**Expected behavior**
the test should pass and it should fail the creation of the `StructArray`

**Additional context**
changing this behavior will also require updating `ArrayData::new_null` which pass nulls for the children of structs.

Also, I did not found any restriction in the arrow specification about that.

But you can see that in the cpp implementation the struct builder append default value for child arrays:
https://github.com/apache/arrow/blob/222fac73a0425e6c273ba8e0b5a427091ee56245/cpp/src/arrow/array/builder_nested.h#L776-L781

and the matching Jira Ticket - [ARROW-9304](https://issues.apache.org/jira/browse/ARROW-9304)

Contributor guide

Open the contributing guide

Research direction

Start with StructArray::try_new and the reproduction test in the issue, then inspect ArrayData::new_null, which the report identifies as related. Compare the current behavior with the mailing-list discussion, Arrow specification, and referenced C++ builder behavior. Done means the reported non-nullable child/null-parent case is rejected and existing null construction remains covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.