[Epic] A Collection of Sort Based Optimizations
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
## Usecase
Many analytic systems store their data with some particular sort order, and the query engine can often take advantage of this sort order to both reduce memory usage and performance
Specific examples in Datafusion include:
1. Emitting from [GroupBy](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/aggregates/order/partial.rs) early with partially sorted stream
2. [`SortMergeJoin`](https://github.com/apache/datafusion/blob/22311835bc1b4bd83b50e1c3875b0e725622b872/datafusion/physical-plan/src/joins/sort_merge_join.rs#L62-L86)
3. Sort removal via [`EnforceSorting`](https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/index.html) and [`replace_with_order_preserving_variants`](https://docs.rs/datafusion/latest/datafusion/physical_optimizer/replace_with_order_preserving_variants/index.html)
This information is currently encoded in [`ExecutionPlan::maintains_input_order`](https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.maintains_input_order) [`ExecutionPlan::required_input_ordering`](https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.required_input_ordering) and [`PlanProperties`](https://docs.rs/datafusion/latest/datafusion/physical_plan/struct.PlanProperties.html)
The same underlying analysis is often required for streaming (where determining what to emit is modeled as a sorted stream, for example on `date_trunc(ts)` of a stream sorted by timestamp).
## Describe the solution you'd like
This epic has a list of optimizations / improvements that further take sortedness into account. Here are some related issues:
- [x] https://github.com/apache/datafusion/issues/4169
- [ ] https://github.com/apache/datafusion/issues/10336
- [x] https://github.com/apache/datafusion/issues/10488
- [ ] https://github.com/apache/datafusion/issues/6672
- [x] https://github.com/apache/datafusion/issues/7490
- [ ] https://github.com/apache/datafusion/issues/4177
- [x] https://github.com/apache/datafusion/issues/10257
- [ ] https://github.com/apache/datafusion/issues/10314
- [ ] https://github.com/apache/datafusion/issues/10316
- [x] #7871
- [x] #13642
- [ ] #7053
- [ ] #14748 (depends on #7053)
Contributor guide
Research direction
Start by reading ExecutionPlan::maintains_input_order, ExecutionPlan::required_input_ordering, PlanProperties, and the linked physical-plan modules for GroupBy, SortMergeJoin, EnforceSorting, and replace_with_order_preserving_variants. Choose one unchecked related issue, such as #10336, #6672, #4177, #10314, #10316, #7053, or #14748, and use its scope to define completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100