[C++] Investigate batching filter node output
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
A highly selective filter could emit tiny batches. Running with tiny batches we have a lot of overhead.
On the other hand, merging batches together requires allocation and copies so it comes with overhead of its own.
When things are more stable we should investigate whether there is some threshold where we merge batches together after a filter.
**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-15521) / @westonpace
**Note**: *This issue was originally created as [ARROW-15521](https://issues.apache.org/jira/browse/ARROW-15521). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
No files, tests, or entry points are named. Start by locating the filter node's batching behavior and measure the overhead of tiny batches against merging allocations and copies. Done means identifying and validating a threshold, if one exists, for merging batches after a highly selective filter.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100