Duplication of data occurs when send from device to pulsar with messageid different but timestamp exactly same upto millisec
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
#### Expected behavior
The number of message received in pulsar should be same as the what the total number of message device send.
#### Actual behavior
1. We send the message from our device to pulsar via websocket.
2. On our device log, we had already send out 2 message within time 2022-07-08 09:14:12 and the status response showing ok means the message already successfully send to the pulsar as shown in below screenshot.

4. But when we check into the pulsar topic the duplication of message occurs as shown in below screenshot.

5. We found that the message that duplicate have different messageID but same publish timestamp upto millisec. It is impossible that our device sent 2 messages within 1 milisec so we suspect that the pulsar have occurs one duplication of data.
6. This issue occurs 1/5000 message that we send throughout the test. Do you have any ideas about this issues?
#### System configuration
**Pulsar version**: 2.7.0
**OS**: Window
#### Additional Information
- suspected related to the old issues : https://github.com/apache/pulsar/issues/7490
Contributor guide
Research direction
Reproduce the intermittent duplication reported when devices send messages through WebSocket to Pulsar 2.7.0 on Windows, comparing device send counts with messages in the topic. The issue is resolved when the cause of distinct message IDs sharing a millisecond timestamp is identified and duplicate delivery is prevented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100