apache / apache/fluss

[server] ZooKeeper Removal and Embedded Raft Metadata Backend

Open
#4,014 0 comments 4 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

Fluss currently depends on an external Apache ZooKeeper ensemble for cluster coordination, metadata persistence, sequence IDs, TabletServer liveness, dynamic config, and ACLs. Clients never talk to ZooKeeper (`bootstrap.servers` + Admin RPC only), but **every server deployment** still needs a healthy ZK quorum.

This extra stateful system is a major operational burden:

- A production Fluss cluster cannot start without ZooKeeper (quorum, tick time, session timeout, SASL/ACL, backups).
- ZK failures cascade into coordinator leadership, TabletServer registration, and metadata availability.
- Container, Kubernetes, edge, and single-node developer setups pay for a second service that exists only for coordination.
- Extra network hops and a second failure domain sit on the metadata path.

This matches the published architecture direction: ZooKeeper is transitional and should be replaced by **KvStore for metadata** and **Raft for coordination**. Kafka KRaft and similar embedded-journal designs show the same class of change is viable when the consensus group stays small and the data plane stays out of the quorum.

**Goals**

1. Default deploy is Fluss-only: CoordinatorServers form a Raft group; no ZooKeeper process is required.
2. A supported dual-backend window so existing ZK clusters can keep running and migrate with tools, not a flag day.
3. No change to client APIs, log/KV format, or `bootstrap.servers`.
4. A later major (Phase 6) removes ZK mode after soak. That deletion is **not** part of the first release of this work.

Tracking FIP: **FIP-52**.

### Solution

Replace ZooKeeper with an **embedded Apache Ratis** Raft quorum on CoordinatorServers, plus a RocksDB-backed metadata KvStore.

- Only coordinators are Raft voters. TabletServers are observers: they discover a coordinator from `coordinator.seed.nodes`, register/heartbeat over RPC, and consume a pull-based `FetchMetadata` stream.
- Consensus is KRaft-style commit-apply: the store is mutated only after a log entry is committed. The Raft term is the coordinator epoch.
- Backend selection is exclusive:
- `zookeeper.address` only → ZooKeeper (deprecated)
- `coordinator.seed.nodes` only → Raft
- both keys → fail fast
- neither key → **Raft** (default)
- Public client APIs, binary log/KV format, and `bootstrap.servers` stay unchanged. New operator CLI: `fluss raft *` and `fluss zk-to-raft-*`.
- This issue / FIP covers Phases 1–5 (dual-support, testing, migration tools, docs, Raft as default). **Phase 6 (delete ZooKeeper mode and the shaded ZK dependency) is out of the first merge.**

**Related materials**

- FIP-52: [`FIP-52`](https://cwiki.apache.org/confluence/spaces/FLUSS/pages/449282216/FIP-52+ZooKeeper+Removal+and+Embedded+Raft+Metadata+Backend)
- Implementation branch: `feature/zookeeper-removal-raft`
- User docs: `website/docs/install-deploy/deploying-with-raft.md`, `website/docs/install-deploy/migrate-zk-to-raft.md`

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with FIP-52 and the implementation branch feature/zookeeper-removal-raft, then read the referenced deployment and migration docs: website/docs/install-deploy/deploying-with-raft.md and website/docs/install-deploy/migrate-zk-to-raft.md. This work is done when Phases 1–5 provide dual-backend support, migration tools and documentation, with Raft as the default; Phase 6 removal of ZooKeeper is excluded.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.