Replicate segments in multiple data centers
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Is your feature request related to a problem? Please describe.**
we have a use case where requires data backup across multiple data centers and needs strong consistency. The ideal view is each segment is replicated to three clusters located at three different data centers. pulsar supports using multiple clusters as a large bookie pool but does not provide the selection logic to ensure each data center has at least one replica.
**Describe the solution you'd like**
allow the segment replicas to be placed at each cluster(or data center). this is useful for use cases like banking in case of disaster.
to my understanding, we have the cluster/topic relationship stored in zookeeper so we can probably have the admin api to enter the replication clusters similar to the allowed-clusters. or we can do it as part of the bookie config file so it will take effect at pulsar instance level
**Describe alternatives you've considered**
We also looked into the geo-replication approach but it's async and increases the cost (number of replicas * cost in a single data center)
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
No files or tests are named. Start by examining how Pulsar stores cluster/topic relationships in ZooKeeper and how bookie placement is selected, then compare the proposed admin API and bookie configuration approaches. Done means segment replicas can be assigned across the specified clusters or data centers with strong consistency.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100