apache / apache/arrow-rs

Optimize `BatchCoalescer::push_batch_with_indices`

Open
#8,957 1 comment 0 reactions 1 assignee Claimed by @Rich-T-kid View on GitHub
enhancement performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We currently have `BatchCoalescer:: push_batch_with_filter` which can speed up filtering of multiple batches while targeting a batch_size, without extra copies / allocations (see https://github.com/apache/arrow-rs/pull/8951).

It would be nice to speed up `BatchCoalescer:: push_batch_with_indices` to cover the case of `take` + `concat` as well (joins / repartitions ...)

**Describe the solution you'd like**

This is added, but just uses `take` followed by the coalesce kernel.
We can speed it up / avoid a copy by handling take internally.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.