Apply filters to `RecordBatch` instead of indices in nested loop join
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
> I think batch coalescer won't make this faster as this is buffering everything in memory anyway.
>
> The main idea would be to apply the filters iteratively to the incoming `RecordBatch` instead of the indices, so we have to change the API / implementation a bit more.
@Dandandan mentions in #16443 that we apply filters to the `RecordBatch` when processing incoming batches in nested loop join.
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by tracing how nested loop join processes incoming RecordBatch values and where filters are currently applied to indices; review the discussion in #16443. Done means defining the required API and implementation changes so filters are applied iteratively to the incoming RecordBatch instead.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100