apache / apache/pulsar

deduplication is not working as expected

Open
#11,445 1 comment 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Create pulsar cluster with version 2.7.1
2. enabled topic level policy at broker config. https://pulsar.apache.org/docs/en/pulsar-admin/#topics-1
3. created a topic and set the deduplicationEnabled to true
4. Created a pulsar producer with a static name and set the timeout to 0 second https://pulsar.apache.org/docs/en/cookbooks-deduplication/
5. produce message with same sequenceId and consume
6. But still I am able to produce and consume multiple message with same sequence id.

**Expected behavior**
Broker must not persist two message with same service id

**Screenshots**
NA

**Desktop (please complete the following information):**
Deployed pulsar on AKS

**Additional context**
How can we handle producer timeout if producer always must have the timeout set to 0(means effective no timeout)

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction steps in the issue and the linked Pulsar topic-policy and deduplication documentation, using Pulsar 2.7.1 and the described producer settings. Investigate why messages with the same sequence ID are both persisted and consumed, and clarify the expected behavior when the producer timeout is zero. Done means duplicate messages are not persisted or consumed under the stated configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.