apache / apache/uniffle

[Improvement] [AQE] Introduce new partition data distribution of global order

Open
#312 0 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

In #293, it introduces the new partition data distribution abstraction. And for AQE skew optimization, it introduce the LOCAL ORDER mode to improve the validity of filtering data and speed up the reads.

But I think it may be not effective especially when shuffle-server's memory is too small that will cause fragmentation problems.

So basing on this assumption, I think maybe for some scenarios, global order of single partition data may be a better solution.

### How should we improve?

## Design
1. Introduce the global order mode in current data distribution
2. Do some performance benchmark for local order and global order

## Global Order details

Shuffle server will know the shuffle reading start only when the first reader reaches. That means the sorting will be started in this time, and other readers will be hang until the sorting finish.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the partition data distribution abstraction introduced in #293 and the existing LOCAL ORDER behavior. Implement and benchmark a GLOBAL ORDER mode, then compare local and global order performance; done means the new mode is available and the benchmark results are recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.