apache / apache/uniffle

[Improvement] Support combine operation in map stage.

Open
#1,238 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

When map send buffer to shuffle server, we could reduce data transmission as much as possible through combine operations.

Because most data may be aggregated in the shuffle server, only part of the data can be combined on the map side. , so the proportion of merge may not be very high.

### How should we improve?

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the map stage sends buffers to the shuffle server and how data is aggregated there. Determine where a combine operation could reduce transmission while preserving shuffle behavior; done means the operation is implemented and its effect is verified with relevant tests or measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.