apache / apache/arrow-rs

[Variant] Support bulk-appends in cast_to_variant

Open
#8,323 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

Re

> My biggest comment / suggestion is to consider making the API vectorized (convert the entire Arrow Array) but I think we can do that as a follow on PR

And https://github.com/apache/arrow-rs/pull/8299#discussion_r2334767295 -- that run-end encoding could be handled more easily in a vectorized API.

And https://github.com/apache/arrow-rs/pull/8299#discussion_r2334753988 that suggests an `append_all_rows()` method.

And https://github.com/apache/arrow-rs/pull/8299#discussion_r2334742498 that also wonders about vectorization.

I'll try to give one response that covers them all:

I think it's reasonable to consider adding a bulk append type API, but we have to be cognizant of the limitations and challenges it will face:
* We will need a new trait that knows how to create (and finish!) variant builder instances
* Variant building is inherently row-based, so any builder that ultimately needs to produce a variant array or variant object as its output will have a trivial `append_all_rows` that just calls `append_row` in a loop (like today), in order to recursively build up the fields/elements of the variant it creates.
* The API would be very nice for converting primitive arrays to variant, because they don't need to recurse on anything. Also nice because we could potentially define a specialized impl just for `VariantArrayBuilder`, so we don't have to deal with that new variant builder create+finish trait.
* Casting a list of primitive values is an interesting intermediate case, where one _should_ be able to append all the elements of a given list in one shot. But that _might_ require the new create+finish trait? Or maybe it just needs a second specialization for `ListBuilder`?
* Maybe instead of a no-arg `append_all_rows()`, we should consider a ranged `append_many_rows(start..end)`? One could always pass `..` to request encoding of all rows.

_Originally posted by @scovich in https://github.com/apache/arrow-rs/issues/8299#issuecomment-3272348634_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.