function EFFECTIVELY_ONCE cannot support input topic batching
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
In pulsar function EFFECTIVELY_ONCE semantic, producer attach a sequenceId to output topic for deduplication, the sequenceId consists of ledgerId and entryId. But for batched messages, the share the same entryId so they have same sequenceId, others will become duplicated and dropped, this may cause data loss.
### Solution
maybe put batchIndex in sequenceId
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start by tracing the Pulsar Functions EFFECTIVELY_ONCE path that creates output sequenceId values and how it handles batched messages. Determine how ledgerId, entryId, and batchIndex are represented, then verify that each batched message receives a distinct sequenceId and is not incorrectly deduplicated or dropped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100