apache / apache/pulsar

[enhancement]Avoid consuming duplicated messages in geo-replication with ReplicateSubscriptionState enabled

Open
#19,551 1 comment 1 reaction 0 assignees View on GitHub
Stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

Geo-replication feature helps us a lot to replicate messages from one cluster to another cluster geographically. However, duplicated messages can occur when the subscription state snapshot is not synchronized correctly between the source and target clusters. We are trying to solve this problem.

### Solution

I found the snapshot will only be taken when the consumer acknowledges a message and advance the subscription cursor we have some snapshot in the cache whose messageId is older than the latest one.
However, some latest cursor positions will not be taken into any snapshot and replicated to the remote cluster.
To solve this problem, we want to set up a scheduled task to run once a second, which will sync the latest subscription cursor position even when the consumer failed and will not advance the cursor any further, but the broker is still alive.

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by tracing geo-replication with ReplicateSubscriptionState enabled, especially subscription cursor advancement and snapshot creation or caching. Define the periodic synchronization entry point and verify that the latest cursor position is replicated when consumers stop acknowledging, preventing duplicated messages; add or run tests covering this failure case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.