apache / apache/uniffle

[FEATURE] MR support slow start

Open
#286 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

Currently, MR on RSS don't support slow start. We can support this feature.

### Motivation

Slow start will improve performance. For 1T terasort, 1w map, 1k reduce, the feature will improve 13% performance.

### Describe the solution

Remove the logic that we forbid the slow start.
Modify the logic of reduce, let reduce can fetch the meta data multiple times. The related logic is in the class `RssEventFetcher`
A big question is how to avoid reading too much duplicate data.

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by tracing the slow-start restriction and the reduce-fetch path around the RssEventFetcher class. Determine how reduce can fetch metadata multiple times without reading excessive duplicate data. Done means MR on RSS supports slow start while avoiding the duplicate-data problem described in the issue; the issue provides no specific test file.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.