[Bug] Bookeeper client in broker deadlock, never recovers on its own

Open

#24,321 opened on May 20, 2025 3 comments 0 reactions 0 assignees View on GitHub

type/bug

Dominant language: Java
Stars: 15.3k
Forks: 3.8k
Avg merge: 1d 14h
Merged PRs (30d): 160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### User environment

AWS EKS cluster, 15 nodes (m4.large),
5 bookie, 5 broker, 5 proxy, 3 zookeeper, 1 recovery, 1 toolset.
Backed by EBS PVs.
Pulsar deployed to ArgoCD via Helm,
Pulsar Helm chart 4.0.0 (Pulsar 4.0.3), from https://pulsar.apache.org/charts.

### Issue Description

After sustaining load for several minutes Pulsar halts, no errors show in either broker pods, proxy pods, bookie pods.
It a partitioned topic, with 32 partitions, using Shared subscription on client consumers.
Only producers errors with:

```
time="2025-05-20T12:01:18Z" level=error msg="Failed to create producer at reconnect" error="server error: UnknownError: Failed to load topic within timeout" producerID=11 producer_name=pulsar-3-22 topic="persistent://public/default/mytopic-partition-9"
time="2025-05-20T12:01:18Z" level=info msg="Reconnecting to broker" assignedBrokerURL= delayReconnectTime=1m3.576257881s producerID=11 producer_name=pulsar-3-22 topic="persistent://public/default/mytopic-partition-9"
```

When investigating using
`kubectl exec -it pulsar-toolset-0 -n pulsar -- bin/pulsar-admin brokers healthcheck`

We get:

```
--- An unexpected error occurred in the server ---

Message: Deadlocked threads detected. BookKeeperClientScheduler-OrderedScheduler-0-0(tid=44), BookKeeperClientWorker-OrderedExecutor-1-0(tid=46)

Stacktrace:

java.lang.IllegalStateException: Deadlocked threads detected. BookKeeperClientScheduler-OrderedScheduler-0-0(tid=44), BookKeeperClientWorker-OrderedExecutor-1-0(tid=46)
at org.apache.pulsar.broker.admin.impl.BrokersBase.checkDeadlockedThreads(BrokersBase.java:431)
at org.apache.pulsar.broker.admin.impl.BrokersBase.lambda$healthCheck$29(BrokersBase.java:395)
at java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.thenAccept(Unknown Source)
at org.apache.pulsar.broker.admin.impl.BrokersBase.healthCheck(BrokersBase.java:395)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:359)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
at org.apache.pulsar.broker.web.WebService$FilterInitializer$WaitUntilPulsarServiceIsReadyForIncomingRequestsFilter.doFilter(WebService.java:336)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.servlets.QoSFilter.doFilter(QoSFilter.java:202)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)

Reason:
--- An unexpected error occurred in the server ---

Message: Deadlocked threads detected. BookKeeperClientScheduler-OrderedScheduler-0-0(tid=44), BookKeeperClientWorker-OrderedExecutor-1-0(tid=46)

Stacktrace:

command terminated with exit code 1
```

Only way to workaround way to recover I found was restart of all broker pods
`kubectl rollout restart statefulset/pulsar-broker -n pulsar`

### Error messages

```text
--- An unexpected error occurred in the server ---

Message: Deadlocked threads detected. BookKeeperClientScheduler-OrderedScheduler-0-0(tid=44), BookKeeperClientWorker-OrderedExecutor-1-0(tid=46)

Stacktrace:

Reason:
--- An unexpected error occurred in the server ---

Message: Deadlocked threads detected. BookKeeperClientScheduler-OrderedScheduler-0-0(tid=44), BookKeeperClientWorker-OrderedExecutor-1-0(tid=46)

Stacktrace:

command terminated with exit code 1
```

### Reproducing the issue

1. Deploy Pulsar as described
2. Push load at ~5k EP/s
3. Wait ~5 minutes

### Additional information

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the `pulsar-admin brokers healthcheck` entry point and the reported `BrokersBase.java` locations at lines 395 and 431, then reproduce the deadlock involving the BookKeeperClient scheduler and worker threads. Done requires identifying why the broker stops recovering and demonstrating recovery for the reported producer and partitioned-topic scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack: aws, helm, java, kubernetes
Domain: backend, distributed-systems
Issue type: Bug
Difficulty: 5/5
Estimated time: Over a week
Activity status: Stale
Clarity: Needs clarification
Newbie friendliness: 25/100

[Bug] Bookeeper client in broker deadlock, never recovers on its own

Description

Contributor guide

Research direction

Assessment

Get new issues in your inbox