apache / apache/dubbo

[Bug] The consumer keep executing the reconnection, after all providers are offline

Open
#15,880 11 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Java
Stars
41.6k
Forks
26.4k
Avg merge
15h 13m
Merged PRs (30d)
4

Description

### Pre-check

- [x] I am sure that all the content I provide is in English.

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.

### Apache Dubbo Component

Java SDK (apache/dubbo)

### Dubbo Version

Dubbo Java 3.3.0, OpenJDK 1.8, Springboot 2.7.18

### Steps to reproduce this issue

One consumer instance and one provider instance and nacos server, just shutdown provider instance.
And the consumer keep executing the reconnection, after all providers are shutdown.

### What you expected to happen

Logs:
```
2025-12-18 16:22:03.563 INFO 39120 --- [andler-thread-1] o.a.d.r.e.s.h.HeaderExchangeHandler : [DUBBO] ChannelReadOnly set true for channel: NettyChannel [channel=[id: 0xb52f26a2, L:/172.16.14.122:53220 - R:/172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:03.593 INFO 39120 --- [andler-thread-1] o.a.d.r.e.s.h.HeaderExchangeHandler : [DUBBO] ChannelReadOnly set true for channel: NettyChannel [channel=[id: 0xb52f26a2, L:/172.16.14.122:53220 - R:/172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.188 INFO 39120 --- [ncesChangeEvent] .r.c.e.l.ServiceInstancesChangedListener : [DUBBO] Received instance notification, serviceName: MSEProvider, instances: 0, dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.188 INFO 39120 --- [ncesChangeEvent] .r.c.e.l.ServiceInstancesChangedListener : [DUBBO] 0 unique working revisions: , dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.188 INFO 39120 --- [ncesChangeEvent] .r.c.e.l.ServiceInstancesChangedListener : [DUBBO] Notify service com.cabbage.demo.mse.common.api.UserService:default:null with urls 1, dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.191 WARN 39120 --- [ncesChangeEvent] .d.r.c.ServiceDiscoveryRegistryDirectory : [DUBBO] Received url with EMPTY protocol from registry ServiceDiscoveryRegistryDirectory(registry: 127.0.0.1:8848, subscribed key: [MSEProvider, MSEServiceProvider, Provider])-Directory(invokers: 1[172.16.14.122:18616], validInvokers: 1[172.16.14.122:18616], invokersToReconnect: 0[]), will clear all available addresses., dubbo version: 3.3.0, current host: 172.16.14.122, error code: 4-1. This may be caused by , go to https://dubbo.apache.org/faq/4/1 to find instructions.
2025-12-18 16:22:04.193 INFO 39120 --- [ncesChangeEvent] o.a.d.r.transport.netty4.NettyChannel : [DUBBO] Close netty channel [id: 0x6733d82a, L:/172.16.14.122:53239 - R:/172.16.14.122:18616], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.197 INFO 39120 --- [ncesChangeEvent] .r.c.m.DefaultMigrationAddressComparator : [DUBBO] No instance address available, stop compare., dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.197 INFO 39120 --- [ncesChangeEvent] .d.r.c.ServiceDiscoveryRegistryDirectory : [DUBBO] Received invokers changed event from registry. Registry type: instance. Service Key: com.cabbage.demo.mse.common.api.UserService:default. Urls Size : 1. Invokers Size : 0. Available Size: 0. Available Invokers : empty, dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:07.203 INFO 39120 --- [lientWorker-5-1] o.a.d.r.t.netty4.NettyConnectionHandler : [DUBBO] **The client has been closed and will not reconnect**. , dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:07.204 INFO 39120 --- [lientWorker-5-1] o.a.d.r.t.netty4.NettyConnectionHandler : [DUBBO] The connection [id: 0x6733d82a, L:/172.16.14.122:53239 ! R:/172.16.14.122:18616] of 172.16.14.122:53239 -> 172.16.14.122:18616 is disconnected., dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:13.527 INFO 39120 --- [lientWorker-4-1] o.a.d.r.t.netty4.NettyClientHandler : [DUBBO] The connection [id: 0xb52f26a2, L:/172.16.14.122:53220 ! R:/172.16.14.122:18616] of 172.16.14.122:53220 -> 172.16.14.122:18616 is disconnected., dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:28.839 INFO 39120 --- [onnect-thread-1] o.a.d.r.e.s.header.ReconnectTimerTask : [DUBBO] Initial connection to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/172.16.14.122:53220 -> /172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:28.840 INFO 39120 --- [onnect-thread-1] o.a.d.r.transport.netty4.NettyChannel : [DUBBO] Close netty channel [id: 0xb52f26a2, L:/172.16.14.122:53220 ! R:/172.16.14.122:18616], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:30.892 ERROR 39120 --- [onnect-thread-1] o.a.d.r.transport.netty4.NettyClient : [DUBBO] Failed to connect to provider server by other reason., dubbo version: 3.3.0, current host: 172.16.14.122, error code: 6-1. This may be caused by network disconnected, go to https://dubbo.apache.org/faq/6/1 to find instructions.

io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /172.16.14.122:18616
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_472]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[na:1.8.0_472]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at java.lang.Thread.run(Thread.java:750) [na:1.8.0_472]

2025-12-18 16:22:30.894 ERROR 39120 --- [onnect-thread-1] o.a.d.r.e.s.header.ReconnectTimerTask : [DUBBO] Fail to connect toHeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/172.16.14.122:53220 -> /172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122, error code: 6-16. This may be caused by , go to https://dubbo.apache.org/faq/6/16 to find instructions.

```
The log tells that **the client has been closed and will not reconnect**, but the consumer keep executing the reconnection.
In the K8s environment, since the provider pod will have a new IP after restart, the consumer keeps reporting errors of attempting to connect to the old IP, which continuously triggers alerts.

### Anything else

_No response_

### Are you willing to submit a pull request to fix on your own?

- [ ] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the reconnection path through ReconnectTimerTask and NettyClient, then follow how ServiceDiscoveryRegistryDirectory handles an empty provider instance list. Reproduce the one-consumer, one-provider shutdown scenario with Dubbo 3.3.0 and Nacos, and verify that removed provider addresses no longer trigger connection attempts or repeated alerts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.