apache / apache/pulsar

[Bug] DLQ and BatchMessages and ConsumerInterceptor don't work well together

Open
#17,871 3 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

2.10.x (and possibly 2.11 and all of the other Pulsar versions), Java Client

### Minimal reproduce step

1. Send batch messages
2. Consume the messages, enabling DLQ, do not acknowledge the messages, set `batchIndexAckEnabled:true`
3. the ConsumerInterceptor#onAckTimeoutSend receives only the EntryId of the each batch (MessageIdImpl, without the batchIndex)

https://github.com/apache/pulsar/blob/99acaab22032f4614072be5d478e785a94f7e492/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerInterceptor.java#L122

### What did you expect to see?

the DLQ processes each message in the batch independently, according to the `batchIndexAckEnabled` policy.
When you set `batchIndexAckEnabled` you expect that every message in the batch is handled as a distinct message.

### What did you see instead?

The messages in a batch are seen as a single message.
The Set of MessageId does NEVER "contain" the MessageIds of the Messages received by the Consumer, there is no way to map them

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerInterceptor.java at onAckTimeoutSend, then reproduce the batch-message flow with DLQ and batchIndexAckEnabled. Trace how the EntryId and MessageId set are passed to the interceptor and DLQ handling. Done means each batch message is represented and processed independently according to the batchIndexAckEnabled policy.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.