apache / apache/arrow-rs

UnionBuilder produces incorrect Union DataType

Open
#1,637 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Describe the bug**
The Union DataType produced by UnionBuilder has non-nullable children Fields after appending nulls in the builder.

**To Reproduce**
Steps to reproduce the behavior: Try the following code
```
let mut builder = UnionBuilder::new_dense(4);
builder.append::("a", 1).unwrap();
builder.append::("b", 3.0).unwrap();
builder.append_null::("b").unwrap();
builder.append_null::("a").unwrap();
let union = builder.build().unwrap();

let schema = Schema::new(vec![
Field::new(
"Teamsters",
DataType::Union(
vec![
Field::new("a", DataType::Int32, true),
Field::new("b", DataType::Float64, true),
],
UnionMode::Dense,
),
false,
),
]);

let batch = RecordBatch::try_new(
Arc::new(schema),
vec![Arc::new(union)]
).unwrap();
```
This code panics:

InvalidArgumentError("column types must match schema types, expected
Union([
Field { name: \"a\", data_type: **Int32, nullable: true**, dict_id: 0, dict_is_ordered: false, metadata: None },
Field { name: \"b\", data_type: **Float64, nullable: true**, dict_id: 0, dict_is_ordered: false, metadata: None }
], Dense
) but found Union([
Field { name: \"a\", data_type: **Int32, nullable: false**, dict_id: 0, dict_is_ordered: false, metadata: None },
Field { name: \"b\", data_type: **Float64, nullable: false**, dict_id: 0, dict_is_ordered: false, metadata: None }
], Dense)
at column index 0")

**Expected behavior**

**Depending on the interpretation of the specification, one of 2 things should happen:**
*A `Union`'s children `Field`s should inherit its nullabillity (i.e. always be false):* Then I think this should error when executing `Field::new()` with a bad `DataType`.

*A child should be nullable if it is capable of returning None to the parent when `unionArray.value(index)` is called*: This code should run just fine then.

**Additional context**
I ran into this when working on #1594. I think it's a simple fix: track the nullablility of the `UnionBuilder` fields rather than always hardcode the child `Field`s nullability to be false. That being said, I'm not sure if that's the correct understanding of the specification.

Contributor guide

Open the contributing guide

Research direction

Start by reading UnionBuilder, its build path, and Field::new, then compare the behavior with the union nullability specification and the context in issue #1594. Reproduce the supplied example and determine which child-field nullability semantics are intended; done means the result follows that decision and the RecordBatch construction no longer fails unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.