apache / apache/pulsar

function EFFECTIVELY_ONCE cannot support input topic batching

Open
#17,061 1 comment 0 reactions 0 assignees View on GitHub
Stale
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

In pulsar function EFFECTIVELY_ONCE semantic, producer attach a sequenceId to output topic for deduplication, the sequenceId consists of ledgerId and entryId. But for batched messages, the share the same entryId so they have same sequenceId, others will become duplicated and dropped, this may cause data loss.

### Solution

maybe put batchIndex in sequenceId

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Pulsar Functions EFFECTIVELY_ONCE path that creates output sequenceId values and how it handles batched messages. Determine how ledgerId, entryId, and batchIndex are represented, then verify that each batched message receives a distinct sequenceId and is not incorrectly deduplicated or dropped.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.