apache / apache/pulsar

[Enhancement] Support replicating already replicated messages and multi-hop forwarding replication topologies

Open
#23,771 6 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

Pulsar replication doesn't replicate already replicated messages:
https://github.com/apache/pulsar/blob/44f986014e4d314a4a52484856c7dfb2d89ea3c1/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/GeoPersistentReplicator.java#L135-L141

Code was refactored in 3.0.0, this is the code in branch-2.11 (same behavior):
https://github.com/apache/pulsar/blob/ff3018afabd0a5ab899ede1441d66dde3d84f68d/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java#L326-L332

It would be useful to have support for multi-hop replication in Pulsar clusters. It would be useful to have support for star (hub-and-spoke) topology as an alternative to full mesh point-to-point topology. Star topology is useful since instead of having every cluster connected with every other cluster, it's possible to configure geo-replication in a star topology where the number of connections is significantly reduced.

This might also have an impact on blue-green cluster migration. The concern is that already replicated messages would get skipped. This issue isn't confirmed.

### Solution

Support multi-hop replication topologies where messages can be forwarded across multiple clusters.

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the referenced replication logic in pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/GeoPersistentReplicator.java and the branch-2.11 PersistentReplicator.java implementation. Trace how already replicated messages are skipped, then determine the changes needed to support forwarding across multiple clusters and star topologies. Done means multi-hop replication works without incorrectly skipping messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.