apache / apache/pulsar

[Bug] Under-configured bookeeper ensemble prevents topics from being created for clients.

Open
#19,080 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

At least 2.9+.

### Minimal reproduce step

1. Set the persistence policy to a namespace such that `--bookkeeper-ensemble` and `--bookkeeper-write-quorum` are 0. Pulsar would accept the values.
```
pulsar-admin namespaces set-persistence public/default --bookkeeper-write-quorum 0 --bookkeeper-ensemble 0 --bookkeeper-ack-quorum 0 --ml-mark-delete-max-rate 0
pulsar-admin namespaces get-persistence public/default
{
"bookkeeperEnsemble" : 0,
"bookkeeperWriteQuorum" : 0,
"bookkeeperAckQuorum" : 0,
"managedLedgerMaxMarkDeleteRate" : 0.0
}
```
2. Create a producer to publish to a new topic in this namespace. The issue is reproducible when there's a need to create a new ledger.
```
publisher = client.newProducer(Schema.STRING).topic("SOME_TOPIC")
.create();
```

3. The producer creation would timeout.
```
2022-12-27T10:58:30,327 - INFO - [pulsar-client-io-1-1:ProducerImpl@1639] - [TOPIC_NAME] [null] Creating producer on cnx [id: 0xc242510c, L:/127.0.0.1:55952 - R:localhost/127.0.0.1:6650]
2022-12-27T10:59:30,065 - ERROR - [pulsar-client-io-1-1:ProducerImpl@1741] - [TOPIC_NAME] [null] Failed to create producer: request timeout {'durationMs': '30000', 'reqId':'316686985597120089', 'remote':'localhost/127.0.0.1:6650', 'local':'/127.0.0.1:55952'}
org.apache.pulsar.client.api.PulsarClientException$TimeoutException: request timeout {'durationMs': '30000', 'reqId':'316686985597120089', 'remote':'localhost/127.0.0.1:6650', 'local':'/127.0.0.1:55952'}
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1008)
at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:88)
```
On the server side, the broker
```
2022-12-27T11:36:42,592-0800 [bookkeeper-ml-scheduler-OrderedScheduler-3-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/default/persistent/TOPIC_NAME
2022-12-27T11:37:42,313-0800 [pulsar-io-19-6] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:56167] Closed producer before its creation was completed. producerId=0
2022-12-27T11:37:42,581-0800 [pulsar-io-19-7] ERROR org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:56167] Failed to create topic persistent://public/default/TOPIC_NAME, producerId=0
java.util.concurrent.CompletionException: org.apache.pulsar.common.util.FutureUtil$LowOverheadTimeoutException: Failed to load topic within timeout
at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368) ~[?:?]
at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377) ~[?:?]
at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1097) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
at org.apache.pulsar.common.util.FutureUtil.lambda$addTimeoutHandling$9(FutureUtil.java:198) ~[pulsar-common.jar:2.12.0-SNAPSHOT]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.pulsar.common.util.FutureUtil$LowOverheadTimeoutException: Failed to load topic within timeout
at org.apache.pulsar.broker.service.BrokerService.futureWithDeadline(...)(Unknown Source) ~[pulsar-broker.jar:2.12.0-SNAPSHOT]
2022-12-27T11:37:42,658-0800 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [public/default/persistent/TOPIC_NAME] Failed to initialize managed ledger: Bookie operation timeout error code: -23
```

### What did you expect to see?

Producers should not be prevented from creation. Or Pulsar should not allow configuration that would result in such situation.

### What did you see instead?

Failure to create topic to the affected namespace.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the persistence-policy path exercised by `pulsar-admin namespaces set-persistence`, then follow producer creation from `ProducerBuilderImpl.create` into `ManagedLedgerImpl`. Reproduce the zero-quorum configuration and inspect the validation and ledger-opening behavior. Done means the invalid configuration is rejected clearly or producers can be created successfully in the affected namespace.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.