apache / apache/datafusion

Vectorize window functions (`GroupsAccumulator`-style)

Open
#15,607 13 comments 8 reactions 0 assignees View on GitHub
performance
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

[More info to be added later]

DataFusion has quite good support for window functions, but (part of) the window function implementation is not yet (I guess mostly because we don't run popular benchmarks with window functions, tpch and clickbench don't have any).

We can try giving it a similar treatment as aggregates (e.g. GroupsAccumulator).

### Describe the solution you'd like

Design a strategy and implement vectorized code for window functions (sum/count, etc) where possible.
Benchmark the improvement.

### Describe alternatives you've considered

_No response_

### Additional context

[Add a profiling image / info here]

Contributor guide

Open the contributing guide

Research direction

Start by reviewing DataFusion’s existing window-function implementation and the GroupsAccumulator-style aggregate approach. Identify which operations, such as sum and count, can be vectorized, then establish benchmark coverage and measure the improvement against the current implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.