apache / apache/pulsar

subscribe to partitioned topic(s) error while non-partitioned topics ok through re.Pattern topics

Open
#12,003 2 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
I tested on subscribing to multiple topics ( partitioned topics error, but non-partitioned topics ok)

**To Reproduce**
Steps to reproduce the behavior:
1. Using pulsar-admin to create one partitioned topic with one partition
2. Using one python consumer to subscribe.
e.g. TOPICS = re.compile(f"persistent://{MQ_TENANT}/{MQ_NAMESPACE}/.*")
3. Error with occur
4. In step 2, if you set your `TOPICS` to subscribe multiple non-partitioned topics seems OK.

**Expected behavior**
By using re.Pattern topics to subscribe should also be OK, just like the non-partitioned topics behavior.

**Screenshots**

```
│ 2021-09-09 23:24:59.043 INFO [140028596160320] ConnectionPool:84 | Created connection for pulsar+ssl://pulsar-proxy.mq:6651
│ 2021-09-09 23:24:59.046 INFO [140028147660544] ClientConnection:372 | [10.0.80.42:47898 -> 172.20.146.245:6651] Connected to broker
│ 2021-09-09 23:24:59.105 INFO [140028147660544] HandlerBase:55 | [persistent://dev/spapi/product_listing_offers-partition-0, spapi.consumer.to_datalake, 0] Getting connection from pool
│r2021-09-09 23:24:59.107 INFO [140028147660544] ConnectionPool:84 | Created connection for pulsar+ssl://pulsar-broker-0.pulsar-broker.mq.svc.cluster.local:6651
│ 2021-09-09 23:24:59.109 INFO [140028147660544] ClientConnection:374 | [10.0.80.42:47902 -> 172.20.146.245:6651] Connected to broker through proxy. Logical broker: pulsar+ssl://pulsar-broker-0.pulsar-broker.mq.svc.cluster.local:6651
│ 2021-09-09 23:24:59.161 WARN [140028147660544] ClientConnection:980 | [10.0.80.42:47902 -> 172.20.146.245:6651] Received error response from server: BrokerPersistenceError (org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available) -- req_id: 0
│ 2021-09-09 23:24:59.161 ERROR [140028147660544] ConsumerImpl:263 | [persistent://dev/spapi/product_listing_offers-partition-0, spapi.consumer.to_datalake, 0] Failed to create consumer: BrokerPersistenceError
│ 2021-09-09 23:24:59.161 ERROR [140028147660544] MultiTopicsConsumerImpl:91 | Failed when subscribed to topic persistent://dev/spapi/product_listing_offers in TopicsConsumer. Error - BrokerPersistenceError
│ 2021-09-09 23:24:59.161 ERROR [140028147660544] MultiTopicsConsumerImpl:101 | Unable to create Consumer - [Muti Topics Consumer: TopicName - persistent://dev/spapi/.* - Subscription - spapi.consumer.to_datalake] Error - BrokerPersistenceError
2021-09-09 23:24:59.161 ERROR [140028147660544] MultiTopicsConsumerImpl:412 | Closing the consumer failed for partition - persistent://dev/spapi/product_listing_offers-partition-0 with error - AlreadyClosed
```

**Desktop (please complete the following information):**
- Base Docker Image: python:3.8.12-slim-buster (Alpine 9.2.0)

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

Reproduce the report by creating a one-partition topic with pulsar-admin and subscribing from a Python consumer using the shown re.Pattern expression. Compare the partitioned-topic failure with subscriptions to multiple non-partitioned topics, and inspect the subscription entry point behind this pattern-based consumer flow. Done means partitioned and non-partitioned topics both subscribe successfully without the reported BrokerPersistenceError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.