apache / apache/uniffle

[Improvement] support sequential unique block id

Open
#1,399 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

The problem of block id overflow is described in #731, #1398.
The block id is used to verify whether the accurate block set is obtained. I think we can get the sequence id from shuffle server. It will be almost impossible for overflow to occur.
In map side, we can generate local block id. Then we report shuffle result. Shuffle will map the local block id to global sequential unique block id. In reduce side, we get shuffle result, then get the global sequential unique block id set.
Since block id is sequential, we only need to pass the length of the bock set.

### How should we improve?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading the related issues #731 and #1398, then trace the map-side, shuffle-server, and reduce-side block ID flow described here. Define how local IDs become globally sequential IDs and how the reduce side represents the block set; completion requires an agreed design and corresponding implementation coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.