apache / apache/pulsar

A host can't connect and won't try other hosts again in multi-hosts service url configuration

Open
#4,510 8 comments 2 reactions 0 assignees View on GitHub
area/client lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Describe the bug**

Use "pulsar://host1:6650,host2:6650,127.0.0.1:6650" as service url to create a pulsar client, "host1" and "host2" is the wrong hosts, not every time I can successfully connect to "127.0.0.1:6650".

**To Reproduce**
```
public void testRetryWithMultiHostServiceUrl() throws PulsarClientException {
String serviceUrl = "pulsar://host1:6650,host2:6650,127.0.0.1:" + BROKER_PORT;
PulsarClient client = PulsarClient.builder().serviceUrl(serviceUrl).build();
Producer producer = client.newProducer().topic("persistent://my-property/my-ns/multi-host").create();
assertNotNull(producer);
}
```
Sometimes will get "java.net.UnknownHostException" and sometimes will connect success.
Here is the stack trace while create producer failed:
```
org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'host1' after 6 queries

at org.apache.pulsar.client.impl.ConnectionPool.lambda$null$9(ConnectionPool.java:202)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'host1' after 6 queries
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.apache.pulsar.client.impl.ConnectionPool.lambda$resolveName$16(ConnectionPool.java:259)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:856)
at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:809)
at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:332)
at io.netty.resolver.dns.DnsResolveContext.onResponse(DnsResolveContext.java:495)
at io.netty.resolver.dns.DnsResolveContext.access$400(DnsResolveContext.java:62)
at io.netty.resolver.dns.DnsResolveContext$3.operationComplete(DnsResolveContext.java:376)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103)
at io.netty.resolver.dns.DnsQueryContext.setSuccess(DnsQueryContext.java:196)
at io.netty.resolver.dns.DnsQueryContext.finish(DnsQueryContext.java:188)
at io.netty.resolver.dns.DnsNameResolver$DnsResponseHandler.channelRead(DnsNameResolver.java:1149)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
... 3 more
Caused by: java.net.UnknownHostException: failed to resolve 'host1' after 6 queries
at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:848)
... 32 more

```
**Expected behavior**

Pulsar client should try to connect to each hosts, only all the hosts can't connect successfully return the error message to users.

Contributor guide

Open the contributing guide

Research direction

Start with ConnectionPool.java, especially the failure paths around lines 202 and 259, and run the provided testRetryWithMultiHostServiceUrl reproduction. Trace how failures for host1 and host2 affect attempts to reach 127.0.0.1. Done means the client tries each configured host and reports an error only when all hosts fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.