apache / apache/fluss

[server] Store large rebalance plans outside ZooKeeper

Open
#3,868 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

The complete rebalance execution plan is currently serialized into a single ZooKeeper node. Large clusters may generate a plan whose encoded size exceeds ZooKeeper's configured payload limit, commonly around 1 MB. The initial plan write, cancellation update, or final-status update can then fail, leaving rebalance unavailable even though the plan itself is valid.

### Proposed solution

Store the immutable rebalance plan in Fluss remote storage, following a lifecycle similar to the KV snapshot lease metadata where appropriate. Keep only compact coordination metadata in ZooKeeper, including:

- Rebalance ID and storage-format version.
- A reference to the remote plan object.
- Integrity information such as the expected size or checksum.
- Overall execution status and cancellation intent.

The publication and recovery protocol should guarantee that:

1. The remote plan is written completely before its reference becomes visible in ZooKeeper.
2. Coordinator restart can resolve and validate the referenced plan before accepting another rebalance.
3. Status and cancellation updates do not rewrite the large plan.
4. Legacy ZooKeeper records containing inline plans remain readable during upgrade.
5. Superseded or abandoned plan objects are cleaned up without deleting the active plan.
6. A missing or corrupt remote plan fails safely and does not allow a conflicting rebalance to start.

### Acceptance criteria

- A serialized plan larger than the ZooKeeper node limit can be submitted and recovered.
- Restart and coordinator failover recover the same plan from its ZooKeeper reference.
- Cancellation and finalization only update compact ZooKeeper state.
- Failure injection covers remote upload failure, ZooKeeper publication failure, missing objects, corrupt objects, and cleanup retry.
- Serialization compatibility with existing inline rebalance tasks is covered by tests.

### Non-goals

This task does not change rebalance scheduling concurrency or bucket migration semantics.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the current rebalance plan serialization and ZooKeeper record lifecycle, then compare the KV snapshot lease metadata flow for the proposed remote-storage lifecycle. Use the acceptance criteria to validate large-plan submission, restart and failover recovery, compact cancellation and finalization updates, legacy inline-plan compatibility, failure injection, and safe cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.