apache / apache/arrow-rs

Support Union types in `StructBuilder`

Open
#6,349 3 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm trying to use `StructBuilder` to create a struct whose one field is of a Union type. This is currently unsupported, see this small repro (crate version v52.2.0):

```rust
use arrow::{
array::StructBuilder,
datatypes::{DataType, Field, Fields, UnionFields, UnionMode},
};

fn main() {
let _builder = StructBuilder::from_fields(
get_struct_fields(),
3,
);
}

fn get_struct_fields() -> Fields {
use arrow::datatypes::{DataType, Field, Fields};
Fields::from(vec![
Field::new("name", DataType::Utf8, false),
Field::new("value", get_union_type(), false),
])
}

fn get_union_type() -> DataType {
let fields = vec![
Field::new("Integer", DataType::Int32, false),
Field::new("Float", DataType::Float32, false),
];
DataType::Union(UnionFields::new(vec![0, 1], fields), UnionMode::Sparse)
}
```

This results in a panic from `builder/struct_builder.rs:278:14`:

```
Data type Union([(0, Field { name: "Integer", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }), (1, Field { name: "Float", data_type: Float32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} })], Sparse) is not currently supported
```

The particular layout of the union doesn't matter, neither does whether it's sparse or dense.

**Describe the solution you'd like**
I'd like to use `StructBuilder` with a Union field the same way it works with other field types. In particular, I should be able to retrieve a `UnionBuilder` for a field:

```
let mut builder = StructBuilder::from_fields(
get_struct_fields(),
3,
);
let union_field = builder.field_builder::(1);
```

**Describe alternatives you've considered**
I don't actually know how to work around this. It'd be appreciated if someone could show me how to build a struct array with a schema like this without using the builder, so I can unblock myself.

**Additional context**
As far as I understand the main issue is that `UnionBuilder` does not implement the `ArrayBuilder` trait, which is required for the dynamic API and internals of `StructBuilder`. This is because `ArrayBuilder` requires a `finish(&mut self) -> ArrayRef`, and the `UnionBuilder` has a `build(self) -> Result` (consumes the builder AND can fail).

Changing `self` to `&mut self` shouldn't be too hard, but the `Result` semantics are rather hard. I'm not sure what the implementation path would be here.

Contributor guide

Open the contributing guide

Research direction

Start at builder/struct_builder.rs:278:14 and trace how StructBuilder requires ArrayBuilder for dynamic fields. Compare that path with UnionBuilder's consuming, fallible build operation; done means the repro no longer panics and a UnionBuilder can be retrieved for the union field.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.