apache / apache/datafusion

Further Optimize "per partition" top-k : `ROW_NUMBER < 5` / TopK

Open
#24,060 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

This ticket track the follow on work from
- https://github.com/apache/datafusion/issues/6899

Where @SubhamSinghal and others have implemented specific optimizations for queries such as this (topK per partition)

```sql
SELECT ...
ROW_NUMBER() OVER (PARTITION BY value1, ORDER BY value2) as rn
WHERE
rn < 10
```

We closed https://github.com/apache/datafusion/issues/6899 to signal we have implemented the first part of the operator; This ticket tracks further optimizations

- [x] https://github.com/apache/datafusion/issues/21594
- [ ] https://github.com/apache/datafusion/issues/21596 / https://github.com/apache/datafusion/pull/23599
- [ ] https://github.com/apache/datafusion/issues/13586

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by reading the prior work in issue 6899, then review unchecked issues 21596 and 13586 and pull request 23599; done means completing the remaining top-k-per-partition optimizations and validating the ROW_NUMBER query behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.