apache / apache/pulsar

Duplication of data occurs when send from device to pulsar with messageid different but timestamp exactly same upto millisec

Open
#16,469 1 comment 0 reactions 0 assignees View on GitHub
Stale
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

#### Expected behavior

The number of message received in pulsar should be same as the what the total number of message device send.

#### Actual behavior

1. We send the message from our device to pulsar via websocket.

2. On our device log, we had already send out 2 message within time 2022-07-08 09:14:12 and the status response showing ok means the message already successfully send to the pulsar as shown in below screenshot.
![image](https://user-images.githubusercontent.com/78590126/177933246-f5ae3775-f362-4d79-8374-d986af6700fd.png)

4. But when we check into the pulsar topic the duplication of message occurs as shown in below screenshot.
![image](https://user-images.githubusercontent.com/78590126/177937052-441256ac-19a0-4df7-bc31-5e3fed3eaab2.png)

5. We found that the message that duplicate have different messageID but same publish timestamp upto millisec. It is impossible that our device sent 2 messages within 1 milisec so we suspect that the pulsar have occurs one duplication of data.

6. This issue occurs 1/5000 message that we send throughout the test. Do you have any ideas about this issues?

#### System configuration
**Pulsar version**: 2.7.0
**OS**: Window

#### Additional Information
- suspected related to the old issues : https://github.com/apache/pulsar/issues/7490

Contributor guide

Open the contributing guide

Research direction

Reproduce the intermittent duplication reported when devices send messages through WebSocket to Pulsar 2.7.0 on Windows, comparing device send counts with messages in the topic. The issue is resolved when the cause of distinct message IDs sharing a millisecond timestamp is identified and duplicate delivery is prevented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.