apache / apache/pulsar

Duplicate message created using partitioned topic with flink

Open
#22,795 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

3.2.3

### Minimal reproduce step

Created a partitioned topic in Apache Pulsar and using Flink to consume it. Flink job has a parallelism of 10. If the JOB is restarted getting duplicate(same) messages

### What did you expect to see?

Need only the unique message

### What did you see instead?

While JOB restart getting same message 5-10 times

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Reproduce the report on Pulsar 3.2.3 with a partitioned topic consumed by a Flink job at parallelism 10, then restart the job and record duplicate deliveries. Compare the observed behavior with the expected unique-message behavior and isolate whether the issue is in Pulsar or Flink before defining a fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.