apache / apache/pulsar

[Bug] Topics appended unlimitedly when subscribing with topics pattern

Open
#20,667 3 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

2.10.2.4

### Minimal reproduce step

Due to consumers keep subscribing to their own retries and dead letter messages, Topic is infinitely appended (`allowAutoTopicCreation = true`)

The code used to reproduce:

```java
public static void main(String[] args) throws IOException {
AuthenticationBasic auth = new AuthenticationBasic();
auth.configure("{\"userId\":\"superuser\",\"password\":\"admin\"}");
String serviceUrl = "pulsar://127.0.0.1:6650";
String topicPattern = "persistent://public/default/ubw.*";
PulsarClient pulsarClient = PulsarClient.builder().serviceUrl(serviceUrl).authentication(auth).build();
var consumerBuilder = pulsarClient.newConsumer(Schema.BYTES).topicsPattern(topicPattern).subscriptionName("ubw-test")
.subscriptionType(SubscriptionType.Shared)
.enableRetry(true)
.deadLetterPolicy(DeadLetterPolicy.builder()
.maxRedeliverCount(3)
.build())
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.messageListener((consumer, msg) -> {
System.out.println("=======================================");
System.out.println("msg = " + new String(msg.getData()));
System.out.println("topic = " + msg.getTopicName());
System.out.println("key = " + msg.getKey());
System.out.println("properties = " + msg.getProperties());
try {
consumer.reconsumeLater(msg, 1, TimeUnit.SECONDS);
} catch (PulsarClientException e) {
throw new RuntimeException(e);
}
}).patternAutoDiscoveryPeriod(1, TimeUnit.SECONDS);
consumerBuilder.subscribe();
System.in.read();
}
```

### What did you expect to see?

This is a bit strange, I think it should automatically acknowledge when the dead letter message is consumed?

This problem will eventually cause Pulsar to throw an exception:
`Got exception org.apache.pulsar.shade.io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5253120: 5263796 - discarded`

### What did you see instead?

Topic is constantly appended, reaching a terrifying length

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the provided Java reproduction and trace topic-pattern discovery alongside retry and dead-letter handling. Determine why generated retry or dead-letter topics are repeatedly included; done means the topic list no longer grows indefinitely and the reported frame-length exception is avoided.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.