apache / apache/pulsar

When one consumer subscribes to multiple topics, disable allowAutoTopicCreation in broker.conf, as long as one topic in the topic list does not exist, messages cannot be received

Open
#11,756 6 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
Pulsar v2.8.0
Disable allowAutoTopicCreation in broker.conf, when one consumer subscribes to multiple topics, as long as one topic in the topic list does not exist, messages cannot be received.

Steps:
1.Create Topic "persistent://public/default/test-string111";
2.Send message to "persistent://public/default/test-string111";
3.Create Consumer:
```
ConsumerBuilder consumerBuilder = getClient().newConsumer(Schema.STRING);
List topics = new ArrayList<>();
topics.add("persistent://public/default/test-string111");
topics.add("persistent://public/default/test-string113");
consumerBuilder.topics(topics)
.subscriptionName("consumerSych")
.subscriptionType(SubscriptionType.Shared)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest);
while (true) {
Message msg = consumer.receive();
consumer.acknowledge(msg);
}
```
Note: Topic "persistent://public/default/test-string113" is not pre created
4.Start Consumer:
```
2021-08-23 18:37:38.472 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string113][consumerSych] Subscribing to topic on cnx [id: 0x2a989e0e, L:/172.32.147.245:9447 - R:/172.32.149.123:16650]
2021-08-23 18:37:38.492 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-0][consumerSych] Subscribing to topic on cnx [id: 0x2a989e0e, L:/172.32.147.245:9447 - R:/172.32.149.123:16650]
2021-08-23 18:37:38.494 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-1][consumerSych] Subscribing to topic on cnx [id: 0x2a989e0e, L:/172.32.147.245:9447 - R:/172.32.149.123:16650]
2021-08-23 18:37:38.495 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-2][consumerSych] Subscribing to topic on cnx [id: 0x2a989e0e, L:/172.32.147.245:9447 - R:/172.32.149.123:16650]
2021-08-23 18:37:38.499 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x2a989e0e, L:/172.32.147.245:9447 - R:/172.32.149.123:16650] Received error from server: Topic does not exist
2021-08-23 18:37:38.503 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string113][consumerSych] Failed to subscribe to topic on /172.32.149.123:16650
2021-08-23 18:37:38.505 [pulsar-client-io-1-1] WARN o.a.pulsar.client.impl.MultiTopicsConsumerImpl - [MultiTopicsConsumer-bcebe] Failed to subscribe for topic [persistent://public/default/test-string113] in topics consumer org.apache.pulsar.client.api.PulsarClientException$TopicDoesNotExistException: Topic does not exist
2021-08-23 18:37:38.511 [pulsar-external-listener-3-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string113] [consumerSych] Closed Consumer (not connected)
2021-08-23 18:37:38.513 [pulsar-external-listener-3-1] WARN o.a.pulsar.client.impl.MultiTopicsConsumerImpl - [MultiTopicsConsumer-bcebe] Failed to subscribe for topic [persistent://public/default/test-string113] in topics consumer, subscribe error: org.apache.pulsar.client.api.PulsarClientException$TopicDoesNotExistException: Topic does not exist
2021-08-23 18:37:38.523 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-2][consumerSych] Subscribed to topic on /172.32.149.123:16650 -- consumer: 3
2021-08-23 18:37:38.524 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-1][consumerSych] Subscribed to topic on /172.32.149.123:16650 -- consumer: 2
2021-08-23 18:37:38.524 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-0][consumerSych] Subscribed to topic on /172.32.149.123:16650 -- consumer: 1
2021-08-23 18:37:38.527 [pulsar-client-io-1-1] WARN o.a.pulsar.client.impl.MultiTopicsConsumerImpl - [MultiTopicsConsumer-bcebe] Failed to subscribe topics: org.apache.pulsar.client.api.PulsarClientException$TopicDoesNotExistException: Topic does not exist
2021-08-23 18:37:38.527 [pulsar-client-io-1-1] INFO o.a.pulsar.client.impl.MultiTopicsConsumerImpl - [MultiTopicsConsumer-bcebe] [consumerSych] Success subscribe new topic persistent://public/default/test-string111 in topics consumer, partitions: 3, allTopicPartitionsNumber: 3
org.apache.pulsar.client.api.PulsarClientException$TopicDoesNotExistException: Topic does not exist
```
5.Consumer can not receive message from persistent://public/default/test-string111;
6.Delete topic "persistent://public/default/test-string113" from consumer code:
```
ConsumerBuilder consumerBuilder = getClient().newConsumer(Schema.STRING);
List topics = new ArrayList<>();
topics.add("persistent://public/default/test-string111");
consumerBuilder.topics(topics)
.subscriptionName("consumerSych")
.subscriptionType(SubscriptionType.Shared)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest);
while (true) {
Message msg = consumer.receive();
consumer.acknowledge(msg);
}
```
7.Start consumer;
8.The consumer still cannot receive messages from the topic "persistent://public/default/test-string111".

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the multi-topic subscription with allowAutoTopicCreation disabled, using ConsumerBuilder and the two topics shown in the issue. Trace the subscription flow through MultiTopicsConsumerImpl and ConsumerImpl, including the TopicDoesNotExistException path. Done means an existing topic still receives messages when another requested topic is absent, with coverage for the reported sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.