Further Optimize "per partition" top-k : `ROW_NUMBER < 5` / TopK
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
This ticket track the follow on work from
- https://github.com/apache/datafusion/issues/6899
Where @SubhamSinghal and others have implemented specific optimizations for queries such as this (topK per partition)
```sql
SELECT ...
ROW_NUMBER() OVER (PARTITION BY value1, ORDER BY value2) as rn
WHERE
rn < 10
```
We closed https://github.com/apache/datafusion/issues/6899 to signal we have implemented the first part of the operator; This ticket tracks further optimizations
- [x] https://github.com/apache/datafusion/issues/21594
- [ ] https://github.com/apache/datafusion/issues/21596 / https://github.com/apache/datafusion/pull/23599
- [ ] https://github.com/apache/datafusion/issues/13586
Contributor guide
Research direction
The issue names no files or tests. Start by reading the prior work in issue 6899, then review unchecked issues 21596 and 13586 and pull request 23599; done means completing the remaining top-k-per-partition optimizations and validating the ROW_NUMBER query behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100