[rebalance] Support preferred leader election goal
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
When a TabletServer becomes unavailable, Fluss automatically elects another in-sync replica as the leader.
For example, the initial leader distribution may be:
```text
TS0: 4
TS1: 4
TS2: 4
```
After TS0 becomes unavailable, its leaders are transferred to the remaining in-sync replicas:
```text
TS0: 0
TS1: 6
TS2: 6
```
When TS0 recovers and its replicas rejoin the ISR, the affected buckets are already in the Online state, so Fluss does not automatically elect their original first replicas again.
The leader distribution may therefore remain at 0/6/6.
The existing LEADER_DISTRIBUTION goal is not equivalent to preferred leader election:
- It balances the total number of leaders across TabletServers.
- It does not guarantee that the first replica in each bucket assignment becomes the leader.
- It may select a different globally balanced leader placement.
- It may fall back to replica movement when leadership movement alone cannot satisfy the distribution goal.
Operators currently have no deterministic and leader-only operation to restore leadership to the first replica in each bucket assignment.
### Solution
Introduce a new rebalance goal:
PREFERRED_LEADER_ELECTION
It can be triggered through the existing Admin API:
admin.rebalance(
Collections.singletonList(GoalType.PREFERRED_LEADER_ELECTION));
and through Flink SQL:
CALL sys.rebalance('PREFERRED_LEADER_ELECTION');
For each bucket:
1. Define the preferred leader as the first replica in the persisted replica assignment, assignment[0].
2. If the current leader is already the preferred leader, do nothing.
3. Otherwise, generate a leader-only rebalance plan when the preferred replica:
- is alive;
- is online;
- is in the current ISR;
- is not tagged TEMPORARY_OFFLINE or PERMANENT_OFFLINE.
4. Elect exactly the preferred replica as the target leader.
5. Do not fall back to another replica if the preferred replica becomes ineligible.
6. Do not change the replica set or perform replica movement.
Buckets whose preferred replica is not currently eligible should remain unchanged and must not be reported as successfully migrated to the preferred leader.
### Anything else?
### Goal interaction
For the first phase, PREFERRED_LEADER_ELECTION should only be accepted as a standalone goal.
For example, the following request should be rejected:
CALL sys.rebalance(
'LEADER_DISTRIBUTION,PREFERRED_LEADER_ELECTION'
);
LEADER_DISTRIBUTION and PREFERRED_LEADER_ELECTION can have conflicting objectives: one optimizes global leader counts, while the other deterministically restores each bucket to
assignment[0].
### Scope
The first phase is intentionally cluster-wide and manually triggered.
The following are not included in this issue:
- automatic preferred leader election after TabletServer recovery;
- periodic detection or automatic failback;
- table-, partition-, or bucket-level filtering;
- replica reassignment;
- under-replicated bucket repair;
- leader load or traffic-based optimization.
These capabilities can be introduced separately later.
### Acceptance criteria
- A bucket whose current leader differs from assignment[0] is moved to assignment[0] when that replica is alive, online, and in the ISR.
- A bucket is left unchanged when its preferred replica is unavailable or outside the ISR.
- The election never falls back to a non-preferred replica.
- The replica assignment and replica count remain unchanged.
- The operation uses only leader-only rebalance tasks.
- Completion waits for a successful acknowledgement from the target preferred leader.
- The operation is idempotent: running it again after completion generates no additional leader movement.
- Requests that combine this goal with another rebalance goal are rejected in the first phase.
- Unit and integration tests cover recovery, unavailable preferred replicas, offline tags, idempotency, and assignment preservation.
### Related work
- #1396 mentioned PreferredLeaderElectionGoal during the early rebalance POC, but it was not included in the final public goals.
- #1452 introduced the rebalance plan generation and execution framework.
- #3071 made leader-only rebalance tasks execute sequentially.
- #3556 requires acknowledgement from the target leader before completing a leader-only task.
- #3869 ensures that replica reassignment applies the leader selected by the generated plan. It does not generate preferred leader plans based on assignment[0].
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the existing Admin API rebalance entry point and Flink SQL CALL sys.rebalance path, then trace GoalType handling, plan generation, and sequential leader-only task execution described in the related rebalance work. Add the standalone PREFERRED_LEADER_ELECTION goal and cover the listed recovery, eligibility, idempotency, acknowledgement, assignment-preservation, and goal-combination acceptance criteria with unit and integration tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100