[Bug] readOffload NoClassDefFoundError
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
4.0.1
### Minimal reproduce step
It can't reproduce on demand, but when a broker has the issue, it always has it.
### What did you expect to see?
Nothing
### What did you see instead?
```
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,810+0000 [offloader-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader - Failed readOffloaded:
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: ... 9 more
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,811+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Unknown exception for ManagedLedgerException.
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: ... 9 more
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,812+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [orga_00575eaa-d956-4b65-90bb-7f87706b3d5a/accesslogs/persistent/app_280d8925-a61e-49b4-8537-1d5748e545e0-partition-0] Error opening ledger for reading at position 162128641:0 - org.apache.bookkeeper.mledger.ManagedLedgerException: Other exception
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,812+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [orga_00575eaa-d956-4b65-90bb-7f87706b3d5a/accesslogs/persistent/app_280d8925-a61e-49b4-8537-1d5748e545e0-partition-0][reader-9bd1511927] read failed from ledger at position:162128641:0
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: org.apache.bookkeeper.mledger.ManagedLedgerException: Other exception
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
```
### Anything else?
Offloaders nar is present. When restarting the broker, the topic is assigned to another broker and it doesn't have the issue now. Also, this error is mixed in many exceptions related to https://github.com/apache/pulsar/issues/23735.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start at BlobStoreManagedLedgerOffloader.java:558, where readOffloaded references BlobStoreBackedReadHandleImpl, and inspect how the offloaders NAR is packaged and loaded. Compare the failure with issue 23735 and validate that offloaded reads no longer produce NoClassDefFoundError after broker restart or reassignment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100