`StructArray` allow to build with child that have nulls but for non nullable field
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**I opened a mailing list discussion on what the behavior should be**
[here](https://lists.apache.org/thread/7gbqjwykh1ob3xbvwph3ljsdl5c7kxpd)
**Describe the bug**
You can create a `StructArray` with field marked as non nullable but the child has nulls.
this should not be possible IMO as:
1. struct field nullability has no meaning if the array can have nulls (I'm aware that the array can only have nulls in the same positions as the struct nulls)
2. you can't have functions asserting non nullable field array has no nulls
3. You can't work on the child arrays in a separate context without the data of the parent
**To Reproduce**
```rust
#[test]
fn reproduction() {
let child = Int8Array::from(vec![None]);
let child = Arc::new(child) as ArrayRef;
let struct_array = StructArray::try_new(
Fields::from(vec![
Field::new(
"f",
child.data_type().clone(),
// Note the false here
false
),
]),
vec![child],
// Creating null item
Some(NullBuffer::from(&[false])),
).expect_err("should fail as field nullability does not match the array nullability");
}
```
**Expected behavior**
the test should pass and it should fail the creation of the `StructArray`
**Additional context**
changing this behavior will also require updating `ArrayData::new_null` which pass nulls for the children of structs.
Also, I did not found any restriction in the arrow specification about that.
But you can see that in the cpp implementation the struct builder append default value for child arrays:
https://github.com/apache/arrow/blob/222fac73a0425e6c273ba8e0b5a427091ee56245/cpp/src/arrow/array/builder_nested.h#L776-L781
and the matching Jira Ticket - [ARROW-9304](https://issues.apache.org/jira/browse/ARROW-9304)
Contributor guide
Research direction
Start with StructArray::try_new and the reproduction test in the issue, then inspect ArrayData::new_null, which the report identifies as related. Compare the current behavior with the mailing-list discussion, Arrow specification, and referenced C++ builder behavior. Done means the reported non-nullable child/null-parent case is rejected and existing null construction remains covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100