[Improvement] support sequential unique block id
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
The problem of block id overflow is described in #731, #1398.
The block id is used to verify whether the accurate block set is obtained. I think we can get the sequence id from shuffle server. It will be almost impossible for overflow to occur.
In map side, we can generate local block id. Then we report shuffle result. Shuffle will map the local block id to global sequential unique block id. In reduce side, we get shuffle result, then get the global sequential unique block id set.
Since block id is sequential, we only need to pass the length of the bock set.
### How should we improve?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Contributor guide
Research direction
Start by reading the related issues #731 and #1398, then trace the map-side, shuffle-server, and reduce-side block ID flow described here. Define how local IDs become globally sequential IDs and how the reduce side represents the block set; completion requires an agreed design and corresponding implementation coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100