apache / apache/arrow-rs

[coalesce] Implement specialized `BatchCoalescer::push_batch` for `StringArray`

Open
#7,764 3 comments 0 reactions 1 assignee Claimed by @Rich-T-kid View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- Part of https://github.com/apache/arrow-rs/issues/7761

The [`BatchCoalescer`](https://github.com/apache/arrow-rs/blob/2b40d1dfc35862ff350a40dfbc66f8a14f4eea31/arrow-select/src/coalesce.rs#L38-L122) 's api `push_batch` incrementally builds up an array and produces a final output

* [`GenericInProgressArray` is a generic implementation](https://github.com/apache/arrow-rs/blob/2b40d1dfc35862ff350a40dfbc66f8a14f4eea31/arrow-select/src/coalesce/generic.rs#L1) that works by buffering `ArrayRef` and then calling `concat`
* There are specialized implementations such as [`InProgressByteViewArray`](https://github.com/apache/arrow-rs/blob/2b40d1dfc35862ff350a40dfbc66f8a14f4eea31/arrow-select/src/coalesce/byte_view.rs#L1) that are more efficient for certain data types (implemented in https://github.com/apache/arrow-rs/pull/7650)

The specialized implementations can go quite a bit faster (30-50% depending)

**Describe the solution you'd like**
Improved performance, as measured by benchmarks for the data type named above

```shell
cargo bench --bench coalesce_kernels
```

**Describe alternatives you've considered**

For `StringArray` and `BinaryArray` the tricky part here will be to avoid copying the data strings as much as possible (by pre-allocating buffer space for example, and postponing the copies appropriately until the required space is known

**Additional context**
- the use case is described in detail here https://github.com/apache/arrow-rs/issues/6692

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.