[FEATURE] MR support slow start
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Currently, MR on RSS don't support slow start. We can support this feature.
### Motivation
Slow start will improve performance. For 1T terasort, 1w map, 1k reduce, the feature will improve 13% performance.
### Describe the solution
Remove the logic that we forbid the slow start.
Modify the logic of reduce, let reduce can fetch the meta data multiple times. The related logic is in the class `RssEventFetcher`
A big question is how to avoid reading too much duplicate data.
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Contributor guide
Research direction
Start by tracing the slow-start restriction and the reduce-fetch path around the RssEventFetcher class. Determine how reduce can fetch metadata multiple times without reading excessive duplicate data. Done means MR on RSS supports slow start while avoiding the duplicate-data problem described in the issue; the issue provides no specific test file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100