apache / apache/pulsar

Use a separated Pulsar client for extensible load manager

Open
#23,350 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

When I'm working on https://github.com/apache/pulsar/pull/23349, I realized that the built-in client should be closed before the load manager. Pulsar provides a built-in client via `PulsarService#getClient()`, which is public and can be used in plugins. It's also used in the built-in topic policies service `SystemTopicBasedTopicPoliciesService`.

Currently, the built-in client was closed after the load manager's close. If there is a pending producer or reader creation from this client, lookup requests will be sent to the load manager, which could fail and the client could retry forever.

For asynchronous creations, there is no way to interrupt them. For example,

```java
// there is no way to interrupt the createAsync() call
final var future = pulsarClient.newProducer().createAsync();
````

The only solution is to close the client. However, since the extensible load manager also uses this client, it would make extensible load manager's cleanup work fail.

### Solution

Create a independent Pulsar client for extensible load manager.

### Alternatives

_No response_

### Anything else?

It should be noted that the built-in Pulsar client's I/O threads config is default. Adding a new built-in client could use more threads.

However, the built-in Pulsar client can still be used without any constraint by plugins.

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by tracing PulsarService#getClient() and the load manager shutdown lifecycle, including SystemTopicBasedTopicPoliciesService usage. Verify how asynchronous producer or reader creation behaves during shutdown, then separate the load manager's client and confirm the load manager can close cleanly without pending built-in-client operations causing lookup retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.