apache / apache/pulsar

[Bug] [client] client with multi-serviceUrl can not connect to server permanently when only one broker shutdown

Open
#20,508 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

master branch. both 2.x, 3.x have this bug

### Minimal reproduce step

In our production environment, only one broker is down, the others are available. However, the clients configured with multi-serviceUrl can not retry connect successfully, continue throwing connectionTimeout exception, finally reach the max retry number and shutdown

Client code such as follow, host1 shutdown, host2 and host3 available, then client can not create producer permanently
`
PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://host1,host2,host3").build();
Producer producer = client.newProducer().topic(topicName).create();
`

The root cause is in PulsarServiceNameResolver#resolveHost. PulsarClient use the same pulsarServiceNameResolver instance to resolveHost when do retry connect. And the retry logic in pulsarServiceNameResolve is roundrobin.

However, if we use pulsarClient to create producer on a partitioned-topic, all the partition share the same pulsarServiceNameResolver. So the retry logic actually is not roundrobin, but random.

The more partitions in topic, the easier this bug occur.

### What did you expect to see?

single point of failure should not exist

### What did you see instead?

single point of failure

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at PulsarServiceNameResolver#resolveHost and trace how retries share the resolver when creating a producer for a partitioned topic. Reproduce with serviceUrl host1,host2,host3 while host1 is unavailable, then verify that the client can create the producer and does not exhaust connection retries while another broker remains available.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.