apache / apache/datafusion

Convert sort to partial_sort when the table is unbounded

Open
#13,807 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

in `fn replace_with_partial_sort`, there is a check
```rust
if !child.execution_mode().is_unbounded() {
return Ok(plan);
}
```
Isn't it better to have partial sorts even if the source is bounded?

### Describe the solution you'd like

Remove this check and always convert sorts to partial sorts if the requirements and properties allow

### Describe alternatives you've considered

_No response_

### Additional context

I guess it would heavily depend on the way how the data is ordered, and what is the rest of the plan, but a few benchmark results perhaps will make it more clear

Contributor guide

Open the contributing guide

Research direction

Start by reading fn replace_with_partial_sort and the surrounding optimizer logic, focusing on the execution-mode check and the requirements and properties it already validates. Compare bounded and unbounded plans, then use or add benchmark coverage to determine when partial_sort is beneficial; done means the conversion is correct for eligible bounded inputs and supported by clear benchmark results.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.