[coalesce] Implement specialized `BatchCoalescer::push_batch` for `StringArray`
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- Part of https://github.com/apache/arrow-rs/issues/7761
The [`BatchCoalescer`](https://github.com/apache/arrow-rs/blob/2b40d1dfc35862ff350a40dfbc66f8a14f4eea31/arrow-select/src/coalesce.rs#L38-L122) 's api `push_batch` incrementally builds up an array and produces a final output
* [`GenericInProgressArray` is a generic implementation](https://github.com/apache/arrow-rs/blob/2b40d1dfc35862ff350a40dfbc66f8a14f4eea31/arrow-select/src/coalesce/generic.rs#L1) that works by buffering `ArrayRef` and then calling `concat`
* There are specialized implementations such as [`InProgressByteViewArray`](https://github.com/apache/arrow-rs/blob/2b40d1dfc35862ff350a40dfbc66f8a14f4eea31/arrow-select/src/coalesce/byte_view.rs#L1) that are more efficient for certain data types (implemented in https://github.com/apache/arrow-rs/pull/7650)
The specialized implementations can go quite a bit faster (30-50% depending)
**Describe the solution you'd like**
Improved performance, as measured by benchmarks for the data type named above
```shell
cargo bench --bench coalesce_kernels
```
**Describe alternatives you've considered**
For `StringArray` and `BinaryArray` the tricky part here will be to avoid copying the data strings as much as possible (by pre-allocating buffer space for example, and postponing the copies appropriately until the required space is known
**Additional context**
- the use case is described in detail here https://github.com/apache/arrow-rs/issues/6692
Contributor guide
Assessment
This issue has not been assessed yet.