[enhancement]Avoid consuming duplicated messages in geo-replication with ReplicateSubscriptionState enabled
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
Geo-replication feature helps us a lot to replicate messages from one cluster to another cluster geographically. However, duplicated messages can occur when the subscription state snapshot is not synchronized correctly between the source and target clusters. We are trying to solve this problem.
### Solution
I found the snapshot will only be taken when the consumer acknowledges a message and advance the subscription cursor we have some snapshot in the cache whose messageId is older than the latest one.
However, some latest cursor positions will not be taken into any snapshot and replicated to the remote cluster.
To solve this problem, we want to set up a scheduled task to run once a second, which will sync the latest subscription cursor position even when the consumer failed and will not advance the cursor any further, but the broker is still alive.
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start by tracing geo-replication with ReplicateSubscriptionState enabled, especially subscription cursor advancement and snapshot creation or caching. Define the periodic synchronization entry point and verify that the latest cursor position is replicated when consumers stop acknowledging, preventing duplicated messages; add or run tests covering this failure case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100