apache / apache/pulsar

[Bug] Offloaders initialization is not blocking

Open
#23,294 5 comments 2 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

3.3.1

### Minimal reproduce step

I have a broker in a cluster of 9 brokers managing 15k namespaces and around 70 000topics distributed on these namespaces and using offload on 50% of them.

Restart a broker.

### What did you expect to see?

Nothing wrong.

### What did you see instead?

```
Sep 12 12:13:00 -pulsar-broker-c3-n9 pulsar[1622527]: 2024-09-12T12:13:00,382+0000 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Unknown exception for ManagedLedgerException.
Sep 12 12:13:00 -pulsar-broker-c3-n9 pulsar[1622527]: java.lang.UnsupportedOperationException: null
Sep 12 12:13:00 -pulsar-broker-c3-n9 pulsar[1622527]: at org.apache.bookkeeper.mledger.impl.NullLedgerOffloader.readOffloaded(NullLedgerOffloader.java:52) ~[org.apache.pulsar-managed-ledger-3.3.1.jar:3.3.1]
```

```
Sep 12 12:09:02 -pulsar-broker-c3-n9 pulsar[1622527]: 2024-09-12T12:09:02,086+0000 [broker-client-shared-internal-executor-5-1] ERROR org.apache.pulsar.broker.PulsarService - create ledgerOffloader failed for namespace orga_8b852f0a-1135-4b4f-8eee-a538f5dd8b64a/pu
lsar_eb900453-73b3-4ebc-8e0e-ef3a38f52ddd83
Sep 12 12:09:02 -pulsar-broker-c3-n9 pulsar[1622527]: org.apache.pulsar.broker.PulsarServerException: java.io.IOException: No offloader found for driver 'aws-s3'. Please make sure you dropped the offloader nar packages under `${PULSAR_HOME}/offloaders`.
Sep 12 12:09:02 -pulsar-broker-c3-n9 pulsar[1622527]: at org.apache.pulsar.broker.PulsarService.createManagedLedgerOffloader(PulsarService.java:1505) ~[org.apache.pulsar-pulsar-broker-3.3.1.jar:3.3.1]
```

### Anything else?

It looks like the broker authorizes managed ledgers actions while the offloaders are not provided (yet).

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with PulsarService.java at createManagedLedgerOffloader and inspect the NullLedgerOffloader.java failure shown in the report. Reproduce a broker restart with the aws-s3 offloader and many namespaces, then trace when managed-ledger actions begin. Done means initialization does not authorize those actions until the configured offloader is available.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.