apache / apache/datafusion

[Epic] A Collection of Sort Based Optimizations

Open
#10,313 6 comments 1 reaction 0 assignees View on GitHub
enhancement PROPOSAL EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Usecase
Many analytic systems store their data with some particular sort order, and the query engine can often take advantage of this sort order to both reduce memory usage and performance

Specific examples in Datafusion include:
1. Emitting from [GroupBy](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/aggregates/order/partial.rs) early with partially sorted stream
2. [`SortMergeJoin`](https://github.com/apache/datafusion/blob/22311835bc1b4bd83b50e1c3875b0e725622b872/datafusion/physical-plan/src/joins/sort_merge_join.rs#L62-L86)
3. Sort removal via [`EnforceSorting`](https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/index.html) and [`replace_with_order_preserving_variants`](https://docs.rs/datafusion/latest/datafusion/physical_optimizer/replace_with_order_preserving_variants/index.html)

This information is currently encoded in [`ExecutionPlan::maintains_input_order`](https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.maintains_input_order) [`ExecutionPlan::required_input_ordering`](https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.required_input_ordering) and [`PlanProperties`](https://docs.rs/datafusion/latest/datafusion/physical_plan/struct.PlanProperties.html)

The same underlying analysis is often required for streaming (where determining what to emit is modeled as a sorted stream, for example on `date_trunc(ts)` of a stream sorted by timestamp).

## Describe the solution you'd like
This epic has a list of optimizations / improvements that further take sortedness into account. Here are some related issues:

- [x] https://github.com/apache/datafusion/issues/4169
- [ ] https://github.com/apache/datafusion/issues/10336
- [x] https://github.com/apache/datafusion/issues/10488
- [ ] https://github.com/apache/datafusion/issues/6672
- [x] https://github.com/apache/datafusion/issues/7490
- [ ] https://github.com/apache/datafusion/issues/4177
- [x] https://github.com/apache/datafusion/issues/10257
- [ ] https://github.com/apache/datafusion/issues/10314
- [ ] https://github.com/apache/datafusion/issues/10316
- [x] #7871
- [x] #13642
- [ ] #7053
- [ ] #14748 (depends on #7053)

Contributor guide

Open the contributing guide

Research direction

Start by reading ExecutionPlan::maintains_input_order, ExecutionPlan::required_input_ordering, PlanProperties, and the linked physical-plan modules for GroupBy, SortMergeJoin, EnforceSorting, and replace_with_order_preserving_variants. Choose one unchecked related issue, such as #10336, #6672, #4177, #10314, #10316, #7053, or #14748, and use its scope to define completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.