Optimize `BatchCoalescer::push_batch_with_indices`
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We currently have `BatchCoalescer:: push_batch_with_filter` which can speed up filtering of multiple batches while targeting a batch_size, without extra copies / allocations (see https://github.com/apache/arrow-rs/pull/8951).
It would be nice to speed up `BatchCoalescer:: push_batch_with_indices` to cover the case of `take` + `concat` as well (joins / repartitions ...)
**Describe the solution you'd like**
This is added, but just uses `take` followed by the coalesce kernel.
We can speed it up / avoid a copy by handling take internally.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.