apache / apache/uniffle

[Improvement] Introduce statestore of shuffle-server for upgrading quickly

Open
#263 19 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

Currently, the cluster of shuffle-servers is hard to update quickly. In the on-premise deployment, we have to write the custom ansible playbook to do rolling update, and the upgrading process is too long.

The rolling update process has two steps:
1. Add the exclude shuffle-servers id into exclude node file which should be recognized by coordinator and no long assigned
2. Waiting the all apps to finished in these excluded shuffle-servers.

Because the shutdown of shuffle-server will lose all app state and cleanup the datafile, we have to handle with care.

If we could store the all state into levelDB or rockdb or localfile(Like Yarn Nodemanager), maybe it will benefit for us to restart quickly and wont make app fail.

### How should we improve?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by tracing the shuffle-server rolling-update flow, including coordinator exclusion handling, application completion waiting, shutdown state loss, and data-file cleanup; define the state that must survive restart before evaluating LevelDB, RocksDB, or local-file storage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.