apache / apache/pulsar

Replicate segments in multiple data centers

Open
#12,980 10 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/feature
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Is your feature request related to a problem? Please describe.**
we have a use case where requires data backup across multiple data centers and needs strong consistency. The ideal view is each segment is replicated to three clusters located at three different data centers. pulsar supports using multiple clusters as a large bookie pool but does not provide the selection logic to ensure each data center has at least one replica.

**Describe the solution you'd like**
allow the segment replicas to be placed at each cluster(or data center). this is useful for use cases like banking in case of disaster.
to my understanding, we have the cluster/topic relationship stored in zookeeper so we can probably have the admin api to enter the replication clusters similar to the allowed-clusters. or we can do it as part of the bookie config file so it will take effect at pulsar instance level
**Describe alternatives you've considered**
We also looked into the geo-replication approach but it's async and increases the cost (number of replicas * cost in a single data center)
**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by examining how Pulsar stores cluster/topic relationships in ZooKeeper and how bookie placement is selected, then compare the proposed admin API and bookie configuration approaches. Done means segment replicas can be assigned across the specified clusters or data centers with strong consistency.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.