apache / apache/datafusion

[EPIC] A collection of `Sort + Limit` / `Top K` optimizations

Open
#7,195 0 comments 1 reaction 0 assignees View on GitHub
enhancement PROPOSAL EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

This ticket has links a collection of various ways to make queries with `LIMIT` or various other variants (like `row_number()` predicates) both:

1. Go faster
2. Use less memory

These are typically called "Top K" style optimizations in databases and optimize the pattern of a sort followed by a limit

```
LIMIT(fetch = 10)
SORT(x)
INPUT...
```

The observation is that if the `INPUT` is much larger than the fetch (aka the `K`) it is much more efficient and less memory intensive to track the top `10` values rather than sort the entire input and discard everything except the top `10`

Normally this done with special `ExecutionPlan` operators. What the operators do and behave depend on the exact query pattern.

### Describe the solution you'd like

- [x] https://github.com/apache/arrow-datafusion/issues/7196
- [x] https://github.com/apache/arrow-datafusion/issues/7149
- [x] https://github.com/apache/arrow-datafusion/issues/6937
- [x] https://github.com/apache/arrow-datafusion/issues/7198
- [x] https://github.com/apache/arrow-datafusion/issues/7064
- [ ] https://github.com/apache/arrow-datafusion/issues/6899
- [x] https://github.com/apache/arrow-datafusion/issues/7191
- [ ] https://github.com/apache/arrow-datafusion/issues/2365
- [x] https://github.com/apache/arrow-datafusion/issues/7162
- [ ] https://github.com/apache/arrow-datafusion/issues/3579
- [x] https://github.com/apache/arrow-datafusion/issues/900

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Use the unchecked linked issues (#6899, #2365, and #3579) as entry points; the checked links identify related work already completed. Start by reading the relevant linked issue and its query pattern, and treat the epic as complete when the listed Top K optimization work is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.