apache / apache/pulsar

[Bug] bin/pulsar-perf will cause the pulsar service to freeze

Open
#23,007 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

OS:Linux 4.19.91-24.8.an8.x86_64 #1 SMP Tue Aug 31 11:30:53 CST 2021 x86_64 x86_64 x86_64 GNU/Linux
Java:java version "17.0.11" 2024-04-16 LTS
Pulsar:v3.0.5

### Minimal reproduce step

1. start pulsar cluster
2. create a persistent topic
3. use bin/pulsar-perf produce tool to produce some produce message, as below
```
bin/pulsar-perf produce \
--admin-url $cluster_host \
--auth-params "token: $token" \
--auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
--num-messages $num \
--num-producers 10 \
--batch-max-messages 100 \
persistent://$tenant_name/$namespace_name/$topic_name
```
5. Lookup request timeout, as below:
6. Worse, this will cause this topic to never produce messages, even if you use the pulsar client SDK
7.

```
java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: Lookup request timeout {'durationMs': '30000', 'reqId':'3305977965004565187', 'remote':'xx.xx.xx.xx/xx.xx.xx.xx:6650', 'local':'/xx.xx.xx.xx:64755'}
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at org.apache.pulsar.testclient.PerformanceProducer.runProducer(PerformanceProducer.java:598) ~[org.apache.pulsar-pulsar-testclient-3.0.5.jar:3.0.5]
at org.apache.pulsar.testclient.PerformanceProducer.lambda$main$1(PerformanceProducer.java:399) ~[org.apache.pulsar-pulsar-testclient-3.0.5.jar:3.0.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: Lookup request timeout {'durationMs': '30000', 'reqId':'3305977965004565187', 'remote':'xx.xx.xx.xx/xx.xx.xx.xx:6650', 'local':'/xx.xx.xx.xx:64755'}
at org.apache.pulsar.client.impl.ClientCnx.checkRequestTimeout(ClientCnx.java:1355) ~[org.apache.pulsar-pulsar-client-original-3.0.5.jar:3.0.5]
at org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:54) ~[org.apache.pulsar-pulsar-common-3.0.5.jar:3.0.5]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:159) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty-netty-transport-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
... 2 more

```

### What did you expect to see?

pulsar can produce and consume messages normally

### What did you see instead?

Pulsar is stuck and times out when producing messages

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Reproduce the timeout with bin/pulsar-perf produce using the reported Pulsar 3.0.5 setup, then inspect org.apache.pulsar.testclient.PerformanceProducer at lines 598 and 399 and the timeout path at ClientCnx.java:1355. Done means identifying and fixing the cause of the lookup timeout and confirming that the topic can produce and consume messages normally afterward.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cli, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.