apache / apache/uniffle

Support client partition data reassign

Open
#1,608 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

# Motivation

After reviewing #1445 again(partition data reassign, which is disabled by default in the master branch), I found some bugs and design problems. I will use this issue to track the further optimizations.

# Subtasks tracking
- [x] #1609
- [x] #1610
- [x] #1583
- [x] #1612
- [x] #1615
- [ ] #1617
- [x] #1652
- [x] #1690
- [x] #1693

# Design thought

## reassign rpc between with spark driver + executor

This part has been involved in the #1445 design doc, I will not describe more.

## reassign signal propagation
In current codebase, the latest reassign partition-> servers plan won't be propagated into the next start tasks.
To solve this problem, I will make writer always get the latest partition->servers plan. Once the reassign signal happens,
the cached shuffleHandleInfo will be updated by the reassign rpc returned.

For the next start task(task2) after reassign tasks finished, task2 will get the latest plan according to the replacement + normal servers list. It will avoid writing to the faulty servers again.

## reassign multiple servers for one partition
This topic is scoped in the single replica.

For the different type partition, we will have different strategies for the partition -> multiple servers assign.
For huge partition, I will hope that after recogizing the huge_partition, we will request reassign multiple servers by rpc, and the task will acheive its owned partitioned server by the hash mechanism by its taskAttemptId,
which will make load balance valid.

For normal partition, the multiple servers are only valid on the reassign multiple times due to the expected problems.
For this case, the task will always get the last server to write.

![image](https://github.com/apache/incubator-uniffle/assets/8609142/75800a5a-dc7e-4e70-b665-bd1fa25075dd)

![image](https://github.com/apache/incubator-uniffle/assets/8609142/2178a7af-b46c-45c6-9e38-2d21d1444eae)

Contributor guide

Open the contributing guide

Research direction

Start by reading the partition data reassign design in #1445 and the completed and outstanding subtasks listed here, especially #1617. Trace how reassign RPC results, shuffle handle information, and partition-to-server plans are propagated between the Spark driver, executors, writers, and subsequent tasks. Done means the remaining optimizations are implemented and the latest server plan is used after reassignment.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.