[Performance] Optimize partial WindowGroupLimit execution
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 22
Description
## Problem
For partial `WindowGroupLimitExec`, where the ranking column is not included in the output, `WindowExec` still creates a regular window processor and materializes a complete ranking array for all input rows.
It then builds a Boolean selection from the ranking array, filters both the ranking array and the input batch, and finally discards the filtered ranking array. This introduces unnecessary allocation and filtering for `row_number`, `rank`, and `dense_rank`.
The selected rows also commonly form contiguous ranges, but the current implementation always uses Arrow filtering.
## Proposed solution
Add a specialized processor for partial `WindowGroupLimitExec` that tracks partition, ordering, and ranking state across input batches and directly produces selected row ranges.
Use zero-copy `RecordBatch::slice` when the selected rows form one contiguous range. For multiple ranges, build the Boolean selection in blocks and use Arrow filtering.
Keep the existing execution path unchanged when window columns are required in the output.
Contributor guide
Research direction
Start at the partial WindowGroupLimitExec and WindowExec execution paths, then trace how row_number, rank, and dense_rank are processed across input batches. Compare the current ranking-array and Arrow-filtering behavior with RecordBatch::slice, and verify that the existing path remains unchanged when window columns are required in the output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100