[Improvement] [AQE] Introduce new partition data distribution of global order
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
In #293, it introduces the new partition data distribution abstraction. And for AQE skew optimization, it introduce the LOCAL ORDER mode to improve the validity of filtering data and speed up the reads.
But I think it may be not effective especially when shuffle-server's memory is too small that will cause fragmentation problems.
So basing on this assumption, I think maybe for some scenarios, global order of single partition data may be a better solution.
### How should we improve?
## Design
1. Introduce the global order mode in current data distribution
2. Do some performance benchmark for local order and global order
## Global Order details
Shuffle server will know the shuffle reading start only when the first reader reaches. That means the sorting will be started in this time, and other readers will be hang until the sorting finish.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Contributor guide
Research direction
Start by reviewing the partition data distribution abstraction introduced in #293 and the existing LOCAL ORDER behavior. Implement and benchmark a GLOBAL ORDER mode, then compare local and global order performance; done means the new mode is available and the benchmark results are recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100