[Bug] Offloaders initialization is not blocking
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
3.3.1
### Minimal reproduce step
I have a broker in a cluster of 9 brokers managing 15k namespaces and around 70 000topics distributed on these namespaces and using offload on 50% of them.
Restart a broker.
### What did you expect to see?
Nothing wrong.
### What did you see instead?
```
Sep 12 12:13:00 -pulsar-broker-c3-n9 pulsar[1622527]: 2024-09-12T12:13:00,382+0000 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Unknown exception for ManagedLedgerException.
Sep 12 12:13:00 -pulsar-broker-c3-n9 pulsar[1622527]: java.lang.UnsupportedOperationException: null
Sep 12 12:13:00 -pulsar-broker-c3-n9 pulsar[1622527]: at org.apache.bookkeeper.mledger.impl.NullLedgerOffloader.readOffloaded(NullLedgerOffloader.java:52) ~[org.apache.pulsar-managed-ledger-3.3.1.jar:3.3.1]
```
```
Sep 12 12:09:02 -pulsar-broker-c3-n9 pulsar[1622527]: 2024-09-12T12:09:02,086+0000 [broker-client-shared-internal-executor-5-1] ERROR org.apache.pulsar.broker.PulsarService - create ledgerOffloader failed for namespace orga_8b852f0a-1135-4b4f-8eee-a538f5dd8b64a/pu
lsar_eb900453-73b3-4ebc-8e0e-ef3a38f52ddd83
Sep 12 12:09:02 -pulsar-broker-c3-n9 pulsar[1622527]: org.apache.pulsar.broker.PulsarServerException: java.io.IOException: No offloader found for driver 'aws-s3'. Please make sure you dropped the offloader nar packages under `${PULSAR_HOME}/offloaders`.
Sep 12 12:09:02 -pulsar-broker-c3-n9 pulsar[1622527]: at org.apache.pulsar.broker.PulsarService.createManagedLedgerOffloader(PulsarService.java:1505) ~[org.apache.pulsar-pulsar-broker-3.3.1.jar:3.3.1]
```
### Anything else?
It looks like the broker authorizes managed ledgers actions while the offloaders are not provided (yet).
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with PulsarService.java at createManagedLedgerOffloader and inspect the NullLedgerOffloader.java failure shown in the report. Reproduce a broker restart with the aws-s3 offloader and many namespaces, then trace when managed-ledger actions begin. Done means initialization does not authorize those actions until the configured offloader is available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100