apache / apache/auron

[Performance] Optimize partial WindowGroupLimit execution

Open
#2,507 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 14h
Merged PRs (30d)
22

Description

## Problem

For partial `WindowGroupLimitExec`, where the ranking column is not included in the output, `WindowExec` still creates a regular window processor and materializes a complete ranking array for all input rows.

It then builds a Boolean selection from the ranking array, filters both the ranking array and the input batch, and finally discards the filtered ranking array. This introduces unnecessary allocation and filtering for `row_number`, `rank`, and `dense_rank`.

The selected rows also commonly form contiguous ranges, but the current implementation always uses Arrow filtering.

## Proposed solution

Add a specialized processor for partial `WindowGroupLimitExec` that tracks partition, ordering, and ranking state across input batches and directly produces selected row ranges.

Use zero-copy `RecordBatch::slice` when the selected rows form one contiguous range. For multiple ranges, build the Boolean selection in blocks and use Arrow filtering.

Keep the existing execution path unchanged when window columns are required in the output.

Contributor guide

Open the contributing guide

Research direction

Start at the partial WindowGroupLimitExec and WindowExec execution paths, then trace how row_number, rank, and dense_rank are processed across input batches. Compare the current ranking-array and Arrow-filtering behavior with RecordBatch::slice, and verify that the existing path remains unchanged when window columns are required in the output.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.