Cluster switching failed for producers

**Describe the bug**

We have deployed two pulsar clusters and enabled geo-replication. We use domain names for load balancing and cluster switching when disaster occurs.
For example, under normal circumstances, producers are connected to cluster A. When cluster A is down, producers can automatically connect to the backup cluster B to perform data operations.

However, in the actual test, it was found that after cluster A was down, even if the domain name was changed and took effect, the producers could not successfully reconnect to the backup cluster B.

**To Reproduce**
Steps to reproduce the behavior:
1. Producers connect to pulsar by pulsar://www.stu13.com:6650 (pulsar://10.187.128.67:6650)
2. Stop pulsar cluster A
3. Modify DNS servers, modify www.stu13.com to 10.187.128.66.
4. The producers keep reconnecting to pulsar://10.187.128.67:6650 and reporting errors

**Expected behavior**
The producers could reconnect to pulsar://10.187.128.66:6650

**Screenshots**
If applicable, add screenshots to help explain your problem.

2021-12-06 11:01:47.293 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xc4441492, L:/10.187.128.66:43650 - R:10.187.128.226/10.187.128.226:6650]] Connected to server
2021-12-06 11:01:47.295 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://gmtp_nasdaq/US_test/metadata] [null] Creating producer on cnx [id: 0xc4441492, L:/10.187.128.66:43650 - R:10.187.128.226/10.187.128.226:6650]
2021-12-06 11:01:47.320 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Created producer on cnx [id: 0xc4441492, L:/10.187.128.66:43650 - R:10.187.128.226/10.187.128.226:6650]
2021-12-06 11:01:47.472 [main] INFO o.s.scheduling.concurrent.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2021-12-06 11:01:47.658 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-4568"]
2021-12-06 11:01:47.711 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 4568 (http) with context path ''
2021-12-06 11:01:47.728 [main] INFO com.gtja.pulsardemo.pulsartest.StartApplication - Started StartApplication in 3.401 seconds (JVM running for 3.863)
2021-12-06 11:01:47.752 [main] INFO com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider initialized
send, msg_key= key_0
send, msg_key= key_1
send, msg_key= key_2
send, msg_key= key_3
2021-12-06 11:02:02.612 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [10.187.128.226/10.187.128.226:6650] Broker notification of Closed producer: 0
2021-12-06 11:02:02.613 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Closed connection [id: 0xc4441492, L:/10.187.128.66:43650 - R:10.187.128.226/10.187.128.226:6650] -- Will try again in 0.1 s
2021-12-06 11:02:02.684 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xc4441492, L:/10.187.128.66:43650 ! R:10.187.128.226/10.187.128.226:6650] Disconnected
2021-12-06 11:02:02.716 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after timeout
2021-12-06 11:02:02.723 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x78f7ae43, L:/10.187.128.66:40850 - R:10.187.128.67/10.187.128.67:6650]] Connected to server
2021-12-06 11:02:02.726 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Creating producer on cnx [id: 0x78f7ae43, L:/10.187.128.66:40850 - R:10.187.128.67/10.187.128.67:6650]
send, msg_key= key_4
2021-12-06 11:02:02.822 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Created producer on cnx [id: 0x78f7ae43, L:/10.187.128.66:40850 - R:10.187.128.67/10.187.128.67:6650]
2021-12-06 11:02:02.823 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Re-Sending 1 messages to server
send, msg_key= key_5
send, msg_key= key_6
send, msg_key= key_7
send, msg_key= key_8
send, msg_key= key_9
send, msg_key= key_10
2021-12-06 11:02:23.365 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [10.187.128.67/10.187.128.67:6650] Broker notification of Closed producer: 0
2021-12-06 11:02:23.366 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Closed connection [id: 0x78f7ae43, L:/10.187.128.66:40850 - R:10.187.128.67/10.187.128.67:6650] -- Will try again in 0.1 s
2021-12-06 11:02:23.468 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after timeout
2021-12-06 11:02:23.475 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ClientCnx - [id: 0xae04dadd, L:/10.187.128.66:40846 - R:www.stu13.com/10.187.128.67:6650] Close connection because received internal-server error No broker was available to own persistent://gmtp_nasdaq/US_test/metadata
2021-12-06 11:02:23.478 [pulsar-client-io-1-1] WARN o.a.pulsar.client.impl.BinaryProtoLookupService - [persistent://gmtp_nasdaq/US_test/metadata] failed to send lookup request : No broker was available to own persistent://gmtp_nasdaq/US_test/metadata
2021-12-06 11:02:23.480 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker was available to own persistent://gmtp_nasdaq/US_test/metadata
2021-12-06 11:02:23.481 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker was available to own persistent://gmtp_nasdaq/US_test/metadata -- Will try again in 0.19 s
2021-12-06 11:02:23.482 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xae04dadd, L:/10.187.128.66:40846 ! R:www.stu13.com/10.187.128.67:6650] Disconnected
2021-12-06 11:02:23.673 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:02:23.677 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x129ed38e, L:/10.187.128.66:40852 - R:www.stu13.com/10.187.128.67:6650]] Connected to server
2021-12-06 11:02:23.682 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ClientCnx - [id: 0x129ed38e, L:/10.187.128.66:40852 - R:www.stu13.com/10.187.128.67:6650] Close connection because received internal-server error No broker was available to own persistent://gmtp_nasdaq/US_test/metadata
2021-12-06 11:02:23.683 [pulsar-client-io-1-1] WARN o.a.pulsar.client.impl.BinaryProtoLookupService - [persistent://gmtp_nasdaq/US_test/metadata] failed to send lookup request : No broker was available to own persistent://gmtp_nasdaq/US_test/metadata
2021-12-06 11:02:23.684 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker was available to own persistent://gmtp_nasdaq/US_test/metadata
2021-12-06 11:02:23.685 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker was available to own persistent://gmtp_nasdaq/US_test/metadata -- Will try again in 0.368 s
2021-12-06 11:02:23.686 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0x129ed38e, L:/10.187.128.66:40852 ! R:www.stu13.com/10.187.128.67:6650] Disconnected
2021-12-06 11:02:23.890 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0x78f7ae43, L:/10.187.128.66:40850 ! R:10.187.128.67/10.187.128.67:6650] Disconnected
send, msg_key= key_11
2021-12-06 11:02:24.055 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:02:24.064 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to www.stu13.com:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:24.066 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:24.066 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650 -- Will try again in 0.773 s
2021-12-06 11:02:24.841 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:02:24.857 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to www.stu13.com:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:24.859 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:24.860 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650 -- Will try again in 1.578 s
2021-12-06 11:02:26.440 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:02:26.443 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to www.stu13.com:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:26.443 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:26.444 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650 -- Will try again in 2.953 s
2021-12-06 11:02:29.398 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:02:29.402 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to www.stu13.com:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:29.403 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:29.405 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650 -- Will try again in 5.794 s
2021-12-06 11:02:35.200 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:02:35.204 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to www.stu13.com:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:35.206 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:35.208 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650 -- Will try again in 12.417 s
2021-12-06 11:02:47.290 [pulsar-timer-5-1] INFO o.a.pulsar.client.impl.ProducerStatsRecorderImpl - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Pending messages: 1 --- Publish throughput: 0.20 msg/s --- 0.00 Mbit/s --- Latency: med: 10.000 ms - 95pct: 44.000 ms - 99pct: 44.000 ms - 99.9pct: 44.000 ms - max: 44.000 ms --- Ack received rate: 0.20 ack/s --- Failed messages: 0
2021-12-06 11:02:47.629 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:02:47.632 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to www.stu13.com:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:47.633 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:47.634 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650 -- Will try again in 5.465 s
2021-12-06 11:02:53.101 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:02:53.105 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to www.stu13.com:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:53.106 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:02:53.106 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650 -- Will try again in 50.946 s
2021-12-06 11:02:53.935 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Message send timed out. Failing 1 messages
2021-12-06 11:02:53.948 [main] INFO o.s.b.a.l.ConditionEvaluationReportLoggingListener -

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-12-06 11:02:53.956 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
java.lang.IllegalStateException: Failed to execute ApplicationRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:789)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:776)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at com.gtja.pulsardemo.pulsartest.StartApplication.main(StartApplication.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: The producer pulsar-cluster-test-84-1 can not send message to the topic persistent://gmtp_nasdaq/US_test/metadata within given timeout : createdAt 30005395867 ns ago, firstSentAt 4668456313504950 ns ago, lastSentAt 4668456313504950 ns ago, retryCount 0
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:961)
at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.send(TypedMessageBuilderImpl.java:91)
at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:63)
at com.gtja.pulsardemo.pulsartest.ProducerDemo.sendMsg(ProducerDemo.java:20)
at com.gtja.pulsardemo.pulsartest.StartApplication.lambda$run$1(StartApplication.java:38)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:786)
... 13 common frames omitted
2021-12-06 11:02:54.057 [main] INFO o.s.scheduling.concurrent.ThreadPoolTaskExecutor - Shutting down ExecutorService 'applicationTaskExecutor'
2021-12-06 11:03:44.054 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:03:44.057 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to www.stu13.com:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:03:44.058 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:03:44.059 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650 -- Will try again in 58.795 s
2021-12-06 11:03:47.292 [pulsar-timer-5-1] INFO o.a.pulsar.client.impl.ProducerStatsRecorderImpl - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Pending messages: 0 --- Publish throughput: 0.00 msg/s --- 0.00 Mbit/s --- Latency: med: 0.000 ms - 95pct: 0.000 ms - 99pct: 0.000 ms - 99.9pct: 0.000 ms - max: -∞ ms --- Ack received rate: 0.00 ack/s --- Failed messages: 1
2021-12-06 11:04:42.857 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:04:42.861 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to www.stu13.com:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:04:42.861 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:04:42.861 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650 -- Will try again in 55.527 s
//
2021-12-06 11:05:38.390 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Reconnecting after connection was closed
2021-12-06 11:05:38.405 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to www.stu13.com:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:05:38.409 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650
2021-12-06 11:05:38.410 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://gmtp_nasdaq/US_test/metadata] [pulsar-cluster-test-84-1] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: 拒绝连接: www.stu13.com/10.187.128.67:6650 -- Will try again in 58.002 s

**Desktop (please complete the following information):**
- OS: [e.g. iOS]

**Additional context**
Add any other context about the problem here.

Start by reproducing the DNS failover steps and compare the reconnecting addresses in the logs. Read the mentioned ConnectionPool, ConnectionHandler, ProducerImpl, ClientCnx, and BinaryProtoLookupService entry points to trace reconnection and lookup behavior. Done means producers reconnect to the backup address after the hostname changes and resume operations.

Written by the indexing model from the issue text.