apache / apache/pulsar

[Bug] readOffload NoClassDefFoundError

Open
#23,736 2 comments 1 reaction 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

4.0.1

### Minimal reproduce step

It can't reproduce on demand, but when a broker has the issue, it always has it.

### What did you expect to see?

Nothing

### What did you see instead?

```
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,810+0000 [offloader-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader - Failed readOffloaded:
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: ... 9 more
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,811+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Unknown exception for ManagedLedgerException.
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: ... 9 more
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,812+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [orga_00575eaa-d956-4b65-90bb-7f87706b3d5a/accesslogs/persistent/app_280d8925-a61e-49b4-8537-1d5748e545e0-partition-0] Error opening ledger for reading at position 162128641:0 - org.apache.bookkeeper.mledger.ManagedLedgerException: Other exception
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,812+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [orga_00575eaa-d956-4b65-90bb-7f87706b3d5a/accesslogs/persistent/app_280d8925-a61e-49b4-8537-1d5748e545e0-partition-0][reader-9bd1511927] read failed from ledger at position:162128641:0
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: org.apache.bookkeeper.mledger.ManagedLedgerException: Other exception
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
```

### Anything else?

Offloaders nar is present. When restarting the broker, the topic is assigned to another broker and it doesn't have the issue now. Also, this error is mixed in many exceptions related to https://github.com/apache/pulsar/issues/23735.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at BlobStoreManagedLedgerOffloader.java:558, where readOffloaded references BlobStoreBackedReadHandleImpl, and inspect how the offloaders NAR is packaged and loaded. Compare the failure with issue 23735 and validate that offloaded reads no longer produce NoClassDefFoundError after broker restart or reassignment.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.