[EPIC] A collection of `Sort + Limit` / `Top K` optimizations
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
This ticket has links a collection of various ways to make queries with `LIMIT` or various other variants (like `row_number()` predicates) both:
1. Go faster
2. Use less memory
These are typically called "Top K" style optimizations in databases and optimize the pattern of a sort followed by a limit
```
LIMIT(fetch = 10)
SORT(x)
INPUT...
```
The observation is that if the `INPUT` is much larger than the fetch (aka the `K`) it is much more efficient and less memory intensive to track the top `10` values rather than sort the entire input and discard everything except the top `10`
Normally this done with special `ExecutionPlan` operators. What the operators do and behave depend on the exact query pattern.
### Describe the solution you'd like
- [x] https://github.com/apache/arrow-datafusion/issues/7196
- [x] https://github.com/apache/arrow-datafusion/issues/7149
- [x] https://github.com/apache/arrow-datafusion/issues/6937
- [x] https://github.com/apache/arrow-datafusion/issues/7198
- [x] https://github.com/apache/arrow-datafusion/issues/7064
- [ ] https://github.com/apache/arrow-datafusion/issues/6899
- [x] https://github.com/apache/arrow-datafusion/issues/7191
- [ ] https://github.com/apache/arrow-datafusion/issues/2365
- [x] https://github.com/apache/arrow-datafusion/issues/7162
- [ ] https://github.com/apache/arrow-datafusion/issues/3579
- [x] https://github.com/apache/arrow-datafusion/issues/900
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Use the unchecked linked issues (#6899, #2365, and #3579) as entry points; the checked links identify related work already completed. Start by reading the relevant linked issue and its query pattern, and treat the epic as complete when the listed Top K optimization work is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100