[Improvement] Make timed-out rebalance tasks recoverable
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
A bucket-level rebalance task currently has a timeout introduced by #3096. When the timeout is reached, the task may be treated as finished even though the replica migration is still in an intermediate state.
For example, a migration from:
```text
[0, 1, 2] -> [1, 0, 3]
```
may time out after reaching:
```text
[0, 1, 2, 3]
```
If the task is removed from tracking at this point, late AdjustIsr or NotifyLeaderAndIsr responses are ignored and the bucket may remain in an intermediate assignment indefinitely. Temporary ZooKeeper failures, RPC timeouts, network fluctuations, coordinator restarts, or TabletServer restarts can all expose this problem.
### Proposed solution
Make timeout a non-final execution state and keep the migration recoverable:
- Remove `TIMEOUT` from the final rebalance statuses.
- Release the normal execution slot after timeout so unrelated pending tasks can continue.
- Continue tracking and periodically reconciling the timed-out task.
- Retry the current migration phase idempotently from persisted assignment and LeaderAndIsr state.
- Avoid increasing the bucket epoch during repeated Phase A retries.
- Fence timeout, reconciliation, and NotifyLeaderAndIsr events by rebalance ID, bucket, attempt ID, coordinator epoch, leader epoch, and bucket epoch.
- Require the target assignment, leader, and ISR to reach the expected state before completion.
- Recover unfinished tasks after coordinator restart or failover.
- Persist cancellation intent before changing local task state, cancel tasks that have not started, and safely drain admitted tasks.
- Do not classify an origin assignment as clean if its ISR still contains replicas introduced by the migration.
- Retry persistence of the final status if the ZooKeeper update fails.
- Treat a bucket as completed if its table or partition is deleted during rebalance.
### Acceptance criteria
- A timed-out task remains visible as non-final and can complete after a late valid response.
- The next pending task can start without losing tracking of the timed-out attempt.
- Repeated Phase A reconciliation does not continuously increase the bucket epoch.
- Stale responses from an older attempt or epoch cannot advance or complete the current task.
- Coordinator restart resumes persisted Phase A, Phase B, cancellation, and finalization states idempotently.
- A dirty origin state such as origin assignment plus an extra ISR replica is not marked canceled.
- A temporarily unavailable target TabletServer can resume the migration after returning with the same server ID.
- Tests cover timeout, late completion, retry, fencing, restart, cancellation, deletion, and TabletServer failure/recovery.
### Non-goals
This task does not cover replacement of a TabletServer that remains offline for a long time. It also does not introduce configurable rebalance concurrency; that work is tracked independently by #3536 and can build on this recovery state machine.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.