[Bug] [client] client with multi-serviceUrl can not connect to server permanently when only one broker shutdown
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
master branch. both 2.x, 3.x have this bug
### Minimal reproduce step
In our production environment, only one broker is down, the others are available. However, the clients configured with multi-serviceUrl can not retry connect successfully, continue throwing connectionTimeout exception, finally reach the max retry number and shutdown
Client code such as follow, host1 shutdown, host2 and host3 available, then client can not create producer permanently
`
PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://host1,host2,host3").build();
Producer producer = client.newProducer().topic(topicName).create();
`
The root cause is in PulsarServiceNameResolver#resolveHost. PulsarClient use the same pulsarServiceNameResolver instance to resolveHost when do retry connect. And the retry logic in pulsarServiceNameResolve is roundrobin.
However, if we use pulsarClient to create producer on a partitioned-topic, all the partition share the same pulsarServiceNameResolver. So the retry logic actually is not roundrobin, but random.
The more partitions in topic, the easier this bug occur.
### What did you expect to see?
single point of failure should not exist
### What did you see instead?
single point of failure
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start at PulsarServiceNameResolver#resolveHost and trace how retries share the resolver when creating a producer for a partitioned topic. Reproduce with serviceUrl host1,host2,host3 while host1 is unavailable, then verify that the client can create the producer and does not exhaust connection retries while another broker remains available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100