[EPIC] Additional improvements to larger than memory / spilling sorts
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Thanks the heroic efforts of @rluvaton @2010YOUY01 @ding-young and many others, the DataFusion spilling sort implementation is quite solid: https://github.com/apache/datafusion/issues/15271
This ticket gathers additional potential improvement for spilling sorts
- [ ] https://github.com/apache/datafusion/issues/14810
- [ ] https://github.com/apache/datafusion/issues/15664
- [ ] https://github.com/apache/datafusion/issues/15321
- [ ] https://github.com/apache/datafusion/issues/15538
- [ ] https://github.com/apache/datafusion/issues/17169
- [ ] https://github.com/apache/datafusion/issues/16908
- [ ] https://github.com/apache/datafusion/issues/14748
Contributor guide
Research direction
Review the seven linked issues, starting with their individual descriptions and any referenced spilling-sort implementation or tests. This epic is complete when the selected linked improvements are implemented and each corresponding issue can be checked off.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100