Support client partition data reassign
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
# Motivation
After reviewing #1445 again(partition data reassign, which is disabled by default in the master branch), I found some bugs and design problems. I will use this issue to track the further optimizations.
# Subtasks tracking
- [x] #1609
- [x] #1610
- [x] #1583
- [x] #1612
- [x] #1615
- [ ] #1617
- [x] #1652
- [x] #1690
- [x] #1693
# Design thought
## reassign rpc between with spark driver + executor
This part has been involved in the #1445 design doc, I will not describe more.
## reassign signal propagation
In current codebase, the latest reassign partition-> servers plan won't be propagated into the next start tasks.
To solve this problem, I will make writer always get the latest partition->servers plan. Once the reassign signal happens,
the cached shuffleHandleInfo will be updated by the reassign rpc returned.
For the next start task(task2) after reassign tasks finished, task2 will get the latest plan according to the replacement + normal servers list. It will avoid writing to the faulty servers again.
## reassign multiple servers for one partition
This topic is scoped in the single replica.
For the different type partition, we will have different strategies for the partition -> multiple servers assign.
For huge partition, I will hope that after recogizing the huge_partition, we will request reassign multiple servers by rpc, and the task will acheive its owned partitioned server by the hash mechanism by its taskAttemptId,
which will make load balance valid.
For normal partition, the multiple servers are only valid on the reassign multiple times due to the expected problems.
For this case, the task will always get the last server to write.


Contributor guide
Research direction
Start by reading the partition data reassign design in #1445 and the completed and outstanding subtasks listed here, especially #1617. Trace how reassign RPC results, shuffle handle information, and partition-to-server plans are propagated between the Spark driver, executors, writers, and subsequent tasks. Done means the remaining optimizations are implemented and the latest server plan is used after reassignment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100