apache / apache/arrow-rs

`ListArray::try_new` rejects `{Union,Dictionary}Array` incorrectly if `field` is not nullable

Open
#6,538 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Describe the bug**
`ListArray::try_new()` uses `Array::is_nullable()` to check the presence of null values if `field` is not marked as nullable.

While this works for most types, this can cause a false positive for `DictionaryArray` and `UnionArray` because `Array::is_nullable()` is allowed to return `true` if it's expensive to prove the absence of logical nulls.

**To Reproduce**

```rust
let offsets = OffsetBuffer::new(vec![0, 1, 4, 5].into());
let mut builder = UnionBuilder::new_dense();
builder.append::("a", 1).unwrap();
builder.append::("b", 2).unwrap();
builder.append::("b", 3).unwrap();
builder.append::("a", 4).unwrap();
builder.append::("a", 5).unwrap();
let values = builder.build().unwrap();
let field = Arc::new(Field::new("element", values.data_type().clone(), false));
ListArray::new(field.clone(), offsets, Arc::new(values), None);
```

This produces:

```
thread 'foo' panicked at foo.rs:10:46:
called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Non-nullable field of ListArray \"element\" cannot contain nulls")
```

**Expected behavior**
Successful execution

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start in arrow-array/src/array/list_array.rs around ListArray::try_new and compare its nullability check with Array::is_nullable in arrow-array/src/array/mod.rs. Run the UnionBuilder reproduction from the issue, then verify that a non-nullable field containing the shown UnionArray is accepted without the false null-error. A regression test should cover the reported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.