Implement spilling for PartialSortExec
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
`PartialSortExec` was added in https://github.com/apache/arrow-datafusion/issues/7456 / https://github.com/apache/arrow-datafusion/pull/9125
While one of the major benefits of this operator is to reduce memory required when sorting data (as it can emit early) we should also handle the case when it still can not fit everything in
### Describe the solution you'd like
Add spilling support to `PartialSortExec` so that if it runs out of memory it will spill to disk rather than error
### Describe alternatives you've considered
_No response_
### Additional context
https://github.com/apache/arrow-datafusion/issues/9153 tracks enabling PartialSort for more queries
Contributor guide
Research direction
Start by locating the PartialSortExec implementation and reading the repository's existing spilling-related code. Identify the relevant tests or add coverage for memory exhaustion; done means PartialSortExec spills to disk instead of returning an out-of-memory error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100