[Bug] The flow rate of some partitions drops to 0 during the pressure test
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
pulsar 2.9.4
### Minimal reproduce step
cannot be reproduced stably
### What did you expect to see?
none
### What did you see instead?
some partitions drops to 0
### Anything else?
When I was doing a stress test, I found that some partitions had no traffic. The client reported `BrokerServiceException$TopicFencedException: Topic is temporarily unavailable`, and the broker reported: 1) `Write of ledger entry to quorum failed: L345593 E11553`, 2)`Failed to persist msg in store: java.util.concurrent.CompletionException: java.lang.NullPointerException` , and later I printed out the specific exception:
```
2023-07-06 20:39:11.0676 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://chenjianbo-tenant/chenjianbo-ns/input-test100-partition-22] Failed to persist msg in store: java.util.concurrent.CompletionException: java.lang.NullPointerException
org.apache.bookkeeper.mledger.ManagedLedgerException$MetaStoreException: java.util.concurrent.CompletionException: java.lang.NullPointerException
Caused by: java.util.concurrent.CompletionException: java.lang.NullPointerException
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_192]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_192]
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:659) ~[?:1.8.0_192]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[?:1.8.0_192]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) ~[?:1.8.0_192]
at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:196) [org.apache.bookkeeper-bookkeeper-common-4.14_vivo-1.1.2.jar:4.14_vivo-1.1.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
Caused by: java.lang.NullPointerException
at org.apache.bookkeeper.mledger.impl.OpAddEntry.createOpAddEntryNoRetainBuffer(OpAddEntry.java:100) ~[org.apache.pulsar-managed-ledger-2.9.2_vivo-1.4-SNAPSHOT.jar:2.9.2_vivo-1.4-SNAPSHOT]
at org.apache.bookkeeper.mledger.impl.OpAddEntry.createNoRetainBuffer(OpAddEntry.java:87) ~[org.apache.pulsar-managed-ledger-2.9.2_vivo-1.4-SNAPSHOT.jar:2.9.2_vivo-1.4-SNAPSHOT]
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.updateLedgersIdsComplete(ManagedLedgerImpl.java:1513) ~[org.apache.pulsar-managed-ledger-2.9.2_vivo-1.4-SNAPSHOT.jar:2.9.2_vivo-1.4-SNAPSHOT]
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$13.operationComplete(ManagedLedgerImpl.java:1430) ~[org.apache.pulsar-managed-ledger-2.9.2_vivo-1.4-SNAPSHOT.jar:2.9.2_vivo-1.4-SNAPSHOT]
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$13.operationComplete(ManagedLedgerImpl.java:1422) ~[org.apache.pulsar-managed-ledger-2.9.2_vivo-1.4-SNAPSHOT.jar:2.9.2_vivo-1.4-SNAPSHOT]
at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$asyncUpdateLedgerIds$6(MetaStoreImpl.java:143) ~[org.apache.pulsar-managed-ledger-2.9.2_vivo-1.4-SNAPSHOT.jar:2.9.2_vivo-1.4-SNAPSHOT]
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) ~[?:1.8.0_192]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[?:1.8.0_192]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) ~[?:1.8.0_192]
at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:196) ~[org.apache.bookkeeper-bookkeeper-common-4.14_vivo-1.1.2.jar:4.14_vivo-1.1.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_192]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_192]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_192]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_192]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_192]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_192]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_192]
```
Contributor guide
Assessment
This issue has not been assessed yet.