`ListArray::try_new` rejects `{Union,Dictionary}Array` incorrectly if `field` is not nullable
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Describe the bug**
`ListArray::try_new()` uses `Array::is_nullable()` to check the presence of null values if `field` is not marked as nullable.
While this works for most types, this can cause a false positive for `DictionaryArray` and `UnionArray` because `Array::is_nullable()` is allowed to return `true` if it's expensive to prove the absence of logical nulls.
**To Reproduce**
```rust
let offsets = OffsetBuffer::new(vec![0, 1, 4, 5].into());
let mut builder = UnionBuilder::new_dense();
builder.append::("a", 1).unwrap();
builder.append::("b", 2).unwrap();
builder.append::("b", 3).unwrap();
builder.append::("a", 4).unwrap();
builder.append::("a", 5).unwrap();
let values = builder.build().unwrap();
let field = Arc::new(Field::new("element", values.data_type().clone(), false));
ListArray::new(field.clone(), offsets, Arc::new(values), None);
```
This produces:
```
thread 'foo' panicked at foo.rs:10:46:
called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Non-nullable field of ListArray \"element\" cannot contain nulls")
```
**Expected behavior**
Successful execution
**Additional context**
Contributor guide
Research direction
Start in arrow-array/src/array/list_array.rs around ListArray::try_new and compare its nullability check with Array::is_nullable in arrow-array/src/array/mod.rs. Run the UnionBuilder reproduction from the issue, then verify that a non-nullable field containing the shown UnionArray is accepted without the false null-error. A regression test should cover the reported behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100