apache / apache/pulsar

PIP-267: Support multi-topic messageId deserialization to ack messages

Open
#20,221 1 comment 0 reactions 1 assignee Claimed by @rdhabalia View on GitHub
Stale type/PIP
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

# Motivation

Pulsar api provides [MessageId](https://github.com/apache/pulsar/blob/master/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/MessageId.java) interface which is generally used by producer and consumer applications to manage topic offset. Sometimes, these applications would like to serialize and deserialize messageIds specifically consumer app which would like to persist messageId and ack with those messageIds by deserializing them. However, right now Pulsar doesn't support correct deserialization of multi-topic or partitioned-topic because of that 1acknowledge` API call fails for those topics with below error
```
org.apache.pulsar.client.api.PulsarClientException$NotAllowedException: Only TopicMessageId is allowed to acknowledge for a multi-topics consumer, while messageId is org.apache.pulsar.client.impl.BatchMessageIdImpl
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1083)
at org.apache.pulsar.client.impl.ConsumerBase.acknowledge(ConsumerBase.java:418)
at org.apache.pulsar.client.api.MultiTopicsConsumerTest.testMultiTopicAckWithByteMessageId(MultiTopicsConsumerTest.java:409)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
```

[MessageIdImpl](https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java#L89) stores id metadata into `MessageIdData` which is doesn't contain context about topic name to find out which topic belongs to this MessageID. Thefore, we need to add topic-name into `MessageIdData` and allow multi-topic/partitioned topic to deserialize message correctly so, API like `acknowledge` can perform as expected.

## Goal

Fix MessageId serialization/deserialization for multi-topic and partitioned-topics so, various apis such as `acknowledge` can work without any error.

This issue is already implemented in this PR: https://github.com/apache/pulsar/pull/19944

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.