[Improvement] Introduce statestore of shuffle-server for upgrading quickly
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
Currently, the cluster of shuffle-servers is hard to update quickly. In the on-premise deployment, we have to write the custom ansible playbook to do rolling update, and the upgrading process is too long.
The rolling update process has two steps:
1. Add the exclude shuffle-servers id into exclude node file which should be recognized by coordinator and no long assigned
2. Waiting the all apps to finished in these excluded shuffle-servers.
Because the shutdown of shuffle-server will lose all app state and cleanup the datafile, we have to handle with care.
If we could store the all state into levelDB or rockdb or localfile(Like Yarn Nodemanager), maybe it will benefit for us to restart quickly and wont make app fail.
### How should we improve?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Contributor guide
Research direction
No files, tests, or entry points are named. Start by tracing the shuffle-server rolling-update flow, including coordinator exclusion handling, application completion waiting, shutdown state loss, and data-file cleanup; define the state that must survive restart before evaluating LevelDB, RocksDB, or local-file storage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100