When one consumer subscribes to multiple topics, disable allowAutoTopicCreation in broker.conf, as long as one topic in the topic list does not exist, messages cannot be received
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
Pulsar v2.8.0
Disable allowAutoTopicCreation in broker.conf, when one consumer subscribes to multiple topics, as long as one topic in the topic list does not exist, messages cannot be received.
Steps:
1.Create Topic "persistent://public/default/test-string111";
2.Send message to "persistent://public/default/test-string111";
3.Create Consumer:
```
ConsumerBuilder consumerBuilder = getClient().newConsumer(Schema.STRING);
List topics = new ArrayList<>();
topics.add("persistent://public/default/test-string111");
topics.add("persistent://public/default/test-string113");
consumerBuilder.topics(topics)
.subscriptionName("consumerSych")
.subscriptionType(SubscriptionType.Shared)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest);
while (true) {
Message msg = consumer.receive();
consumer.acknowledge(msg);
}
```
Note: Topic "persistent://public/default/test-string113" is not pre created
4.Start Consumer:
```
2021-08-23 18:37:38.472 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string113][consumerSych] Subscribing to topic on cnx [id: 0x2a989e0e, L:/172.32.147.245:9447 - R:/172.32.149.123:16650]
2021-08-23 18:37:38.492 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-0][consumerSych] Subscribing to topic on cnx [id: 0x2a989e0e, L:/172.32.147.245:9447 - R:/172.32.149.123:16650]
2021-08-23 18:37:38.494 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-1][consumerSych] Subscribing to topic on cnx [id: 0x2a989e0e, L:/172.32.147.245:9447 - R:/172.32.149.123:16650]
2021-08-23 18:37:38.495 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-2][consumerSych] Subscribing to topic on cnx [id: 0x2a989e0e, L:/172.32.147.245:9447 - R:/172.32.149.123:16650]
2021-08-23 18:37:38.499 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x2a989e0e, L:/172.32.147.245:9447 - R:/172.32.149.123:16650] Received error from server: Topic does not exist
2021-08-23 18:37:38.503 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string113][consumerSych] Failed to subscribe to topic on /172.32.149.123:16650
2021-08-23 18:37:38.505 [pulsar-client-io-1-1] WARN o.a.pulsar.client.impl.MultiTopicsConsumerImpl - [MultiTopicsConsumer-bcebe] Failed to subscribe for topic [persistent://public/default/test-string113] in topics consumer org.apache.pulsar.client.api.PulsarClientException$TopicDoesNotExistException: Topic does not exist
2021-08-23 18:37:38.511 [pulsar-external-listener-3-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string113] [consumerSych] Closed Consumer (not connected)
2021-08-23 18:37:38.513 [pulsar-external-listener-3-1] WARN o.a.pulsar.client.impl.MultiTopicsConsumerImpl - [MultiTopicsConsumer-bcebe] Failed to subscribe for topic [persistent://public/default/test-string113] in topics consumer, subscribe error: org.apache.pulsar.client.api.PulsarClientException$TopicDoesNotExistException: Topic does not exist
2021-08-23 18:37:38.523 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-2][consumerSych] Subscribed to topic on /172.32.149.123:16650 -- consumer: 3
2021-08-23 18:37:38.524 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-1][consumerSych] Subscribed to topic on /172.32.149.123:16650 -- consumer: 2
2021-08-23 18:37:38.524 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-string111-partition-0][consumerSych] Subscribed to topic on /172.32.149.123:16650 -- consumer: 1
2021-08-23 18:37:38.527 [pulsar-client-io-1-1] WARN o.a.pulsar.client.impl.MultiTopicsConsumerImpl - [MultiTopicsConsumer-bcebe] Failed to subscribe topics: org.apache.pulsar.client.api.PulsarClientException$TopicDoesNotExistException: Topic does not exist
2021-08-23 18:37:38.527 [pulsar-client-io-1-1] INFO o.a.pulsar.client.impl.MultiTopicsConsumerImpl - [MultiTopicsConsumer-bcebe] [consumerSych] Success subscribe new topic persistent://public/default/test-string111 in topics consumer, partitions: 3, allTopicPartitionsNumber: 3
org.apache.pulsar.client.api.PulsarClientException$TopicDoesNotExistException: Topic does not exist
```
5.Consumer can not receive message from persistent://public/default/test-string111;
6.Delete topic "persistent://public/default/test-string113" from consumer code:
```
ConsumerBuilder consumerBuilder = getClient().newConsumer(Schema.STRING);
List topics = new ArrayList<>();
topics.add("persistent://public/default/test-string111");
consumerBuilder.topics(topics)
.subscriptionName("consumerSych")
.subscriptionType(SubscriptionType.Shared)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest);
while (true) {
Message msg = consumer.receive();
consumer.acknowledge(msg);
}
```
7.Start consumer;
8.The consumer still cannot receive messages from the topic "persistent://public/default/test-string111".
Contributor guide
Research direction
Start by reproducing the multi-topic subscription with allowAutoTopicCreation disabled, using ConsumerBuilder and the two topics shown in the issue. Trace the subscription flow through MultiTopicsConsumerImpl and ConsumerImpl, including the TopicDoesNotExistException path. Done means an existing topic still receives messages when another requested topic is absent, with coverage for the reported sequence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100