[Bug] Under-configured bookeeper ensemble prevents topics from being created for clients.
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
At least 2.9+.
### Minimal reproduce step
1. Set the persistence policy to a namespace such that `--bookkeeper-ensemble` and `--bookkeeper-write-quorum` are 0. Pulsar would accept the values.
```
pulsar-admin namespaces set-persistence public/default --bookkeeper-write-quorum 0 --bookkeeper-ensemble 0 --bookkeeper-ack-quorum 0 --ml-mark-delete-max-rate 0
pulsar-admin namespaces get-persistence public/default
{
"bookkeeperEnsemble" : 0,
"bookkeeperWriteQuorum" : 0,
"bookkeeperAckQuorum" : 0,
"managedLedgerMaxMarkDeleteRate" : 0.0
}
```
2. Create a producer to publish to a new topic in this namespace. The issue is reproducible when there's a need to create a new ledger.
```
publisher = client.newProducer(Schema.STRING).topic("SOME_TOPIC")
.create();
```
3. The producer creation would timeout.
```
2022-12-27T10:58:30,327 - INFO - [pulsar-client-io-1-1:ProducerImpl@1639] - [TOPIC_NAME] [null] Creating producer on cnx [id: 0xc242510c, L:/127.0.0.1:55952 - R:localhost/127.0.0.1:6650]
2022-12-27T10:59:30,065 - ERROR - [pulsar-client-io-1-1:ProducerImpl@1741] - [TOPIC_NAME] [null] Failed to create producer: request timeout {'durationMs': '30000', 'reqId':'316686985597120089', 'remote':'localhost/127.0.0.1:6650', 'local':'/127.0.0.1:55952'}
org.apache.pulsar.client.api.PulsarClientException$TimeoutException: request timeout {'durationMs': '30000', 'reqId':'316686985597120089', 'remote':'localhost/127.0.0.1:6650', 'local':'/127.0.0.1:55952'}
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1008)
at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:88)
```
On the server side, the broker
```
2022-12-27T11:36:42,592-0800 [bookkeeper-ml-scheduler-OrderedScheduler-3-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/default/persistent/TOPIC_NAME
2022-12-27T11:37:42,313-0800 [pulsar-io-19-6] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:56167] Closed producer before its creation was completed. producerId=0
2022-12-27T11:37:42,581-0800 [pulsar-io-19-7] ERROR org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:56167] Failed to create topic persistent://public/default/TOPIC_NAME, producerId=0
java.util.concurrent.CompletionException: org.apache.pulsar.common.util.FutureUtil$LowOverheadTimeoutException: Failed to load topic within timeout
at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368) ~[?:?]
at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377) ~[?:?]
at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1097) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
at org.apache.pulsar.common.util.FutureUtil.lambda$addTimeoutHandling$9(FutureUtil.java:198) ~[pulsar-common.jar:2.12.0-SNAPSHOT]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.pulsar.common.util.FutureUtil$LowOverheadTimeoutException: Failed to load topic within timeout
at org.apache.pulsar.broker.service.BrokerService.futureWithDeadline(...)(Unknown Source) ~[pulsar-broker.jar:2.12.0-SNAPSHOT]
2022-12-27T11:37:42,658-0800 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [public/default/persistent/TOPIC_NAME] Failed to initialize managed ledger: Bookie operation timeout error code: -23
```
### What did you expect to see?
Producers should not be prevented from creation. Or Pulsar should not allow configuration that would result in such situation.
### What did you see instead?
Failure to create topic to the affected namespace.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start with the persistence-policy path exercised by `pulsar-admin namespaces set-persistence`, then follow producer creation from `ProducerBuilderImpl.create` into `ManagedLedgerImpl`. Reproduce the zero-quorum configuration and inspect the validation and ledger-opening behavior. Done means the invalid configuration is rejected clearly or producers can be created successfully in the affected namespace.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100