[server] Store large rebalance plans outside ZooKeeper
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
The complete rebalance execution plan is currently serialized into a single ZooKeeper node. Large clusters may generate a plan whose encoded size exceeds ZooKeeper's configured payload limit, commonly around 1 MB. The initial plan write, cancellation update, or final-status update can then fail, leaving rebalance unavailable even though the plan itself is valid.
### Proposed solution
Store the immutable rebalance plan in Fluss remote storage, following a lifecycle similar to the KV snapshot lease metadata where appropriate. Keep only compact coordination metadata in ZooKeeper, including:
- Rebalance ID and storage-format version.
- A reference to the remote plan object.
- Integrity information such as the expected size or checksum.
- Overall execution status and cancellation intent.
The publication and recovery protocol should guarantee that:
1. The remote plan is written completely before its reference becomes visible in ZooKeeper.
2. Coordinator restart can resolve and validate the referenced plan before accepting another rebalance.
3. Status and cancellation updates do not rewrite the large plan.
4. Legacy ZooKeeper records containing inline plans remain readable during upgrade.
5. Superseded or abandoned plan objects are cleaned up without deleting the active plan.
6. A missing or corrupt remote plan fails safely and does not allow a conflicting rebalance to start.
### Acceptance criteria
- A serialized plan larger than the ZooKeeper node limit can be submitted and recovered.
- Restart and coordinator failover recover the same plan from its ZooKeeper reference.
- Cancellation and finalization only update compact ZooKeeper state.
- Failure injection covers remote upload failure, ZooKeeper publication failure, missing objects, corrupt objects, and cleanup retry.
- Serialization compatibility with existing inline rebalance tasks is covered by tests.
### Non-goals
This task does not change rebalance scheduling concurrency or bucket migration semantics.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the current rebalance plan serialization and ZooKeeper record lifecycle, then compare the KV snapshot lease metadata flow for the proposed remote-storage lifecycle. Use the acceptance criteria to validate large-plan submission, restart and failover recovery, compact cancellation and finalization updates, legacy inline-plan compatibility, failure injection, and safe cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100