[Bug] Dubbo 3.3.2: FORCE_APPLICATION may leave one of multiple same-interface different-group consumer references with nacos-A(0) after startup when mapping is initially empty
- Dominant language
- Java
- Stars
- 41.6k
- Forks
- 26.4k
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 4
Description
### Pre-check
- [x] I am sure that all the content I provide is in English.
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.
### Apache Dubbo Component
Java SDK (apache/dubbo)
### Dubbo Version
Dubbo: 3.3.2
JDK: openjdk version "1.8.0_452"
OS: SUSE Linux Enterprise Server 15 SP7 (x86_64) - Kernel
Registry: Nacos
Protocol: tri
Migration mode: FORCE_APPLICATION
### Steps to reproduce this issue
### Environment
- Dubbo version: `3.3.2`
- Registry: `Nacos`
- Protocol: `tri`
- Migration mode: `FORCE_APPLICATION`
- One consumer JVM contains multiple references of the same interface but with different groups
- `enable-empty-protection=true`
Consumer configuration:
```properties
dubbo.application.name=mng-consumer
dubbo.application.service-discovery.migration=FORCE_APPLICATION
dubbo.application.shutwait=30000
dubbo.application.enable-empty-protection=true
dubbo.reference.check=false
dubbo.reference.filter=-authenticationPrepare,-contextHolderParametersSelectedTransfer
dubbo.consumer.parameters.params-filter=-authenticationResolver,-authenticationExceptionTranslator
dubbo.consumer.parameters.router=-tag
dubbo.consumer.protocol=tri
dubbo.consumer.timeout=180000
dubbo.protocols.tri.name=tri
dubbo.protocols.tri.triple.max-response-body-size=52428800
dubbo.protocols.tri.triple.max-body-size=52428800
```
### Scenario
In one consumer JVM, there are multiple references of the same interface but with different groups, for example:
- `cbsp-limt1/com.szfesc.cbsp.limt.api.LimitReProcServiceApi`
- `cbsp-limt2/com.szfesc.cbsp.limt.api.LimitReProcServiceApi`
### Reproduction pattern
This issue does not happen every time. It seems to happen when startup enters the path where interface-app mapping is initially empty.
Observed startup log:
```text
No interface-apps mapping found in local cache, stop subscribing, will automatically wait for mapping listener callback:
... group=cbsp-limt1&interface=com.szfesc.cbsp.limt.api.LimitReProcServiceApi ...
```
After that, mapping callback and Nacos app subscription logs can still be observed, for example:
```text
[DUBBO] Received mapping notification from meta server, {serviceKey: com.szfesc.cbsp.limt.api.LimitReProcServiceApi, apps: [cbsp-limt1]}
```
```text
[SUBSCRIBE-SERVICE] service:cbsp-limt1, group:RPC_GROUP, clusters:
new ips(1) service: RPC_GROUP@@cbsp-limt1 -> [...]
```
However, one concrete consumer reference may still remain unavailable in qos output:
```text
As Consumer side:
+---------------------------------------------------------+----------+
| Consumer Service Name | NUM |
+---------------------------------------------------------+----------+
|cbsp-limt1/com.szfesc.cbsp.limt.api.LimitReProcServiceApi|nacos-A(0)|
+---------------------------------------------------------+----------+
|cbsp-limt2/com.szfesc.cbsp.limt.api.LimitReProcServiceApi|nacos-A(2)|
+---------------------------------------------------------+----------+
```
Restarting the consumer process makes it recover.
### Important observation
For the affected reference, I do NOT see the final address notify log:
```text
Notify service cbsp-limt1/com.szfesc.cbsp.limt.api.LimitReProcServiceApi:tri with urls ...
```
This suggests the affected concrete reference may not be successfully attached into the final application-level notify chain, even though mapping callback and Nacos app-level subscription do happen.
### What you expected to happen
I expect that after mapping callback arrives, all concrete consumer references of the same interface in the JVM should eventually complete the application-level subscribe flow and receive address notifications correctly.
In the example above, both references should eventually become non-zero in qos output, instead of one staying at `nacos-A(0)` permanently until restart.
### Anything else
### Why I think this is a Dubbo bug instead of a user configuration problem
- Providers are visible and healthy in Nacos.
- Other references in the same JVM are normal.
- The issue only affects one concrete reference while another reference of the same interface in the same process works.
- Restarting the consumer fixes it.
- This behavior looks like a startup race / recovery issue in application-level service discovery.
### Relevant logs
Startup entered mapping-miss path:
```text
No interface-apps mapping found in local cache, stop subscribing, will automatically wait for mapping listener callback:
... group=cbsp-limt1&interface=com.szfesc.cbsp.limt.api.LimitReProcServiceApi ...
```
Mapping callback was received:
```text
[DUBBO] Received mapping notification from meta server, {serviceKey: com.szfesc.cbsp.limt.api.LimitReProcServiceApi, apps: [cbsp-limt1]}
```
Nacos app-level subscription happened:
```text
[SUBSCRIBE-SERVICE] service:cbsp-limt1, group:RPC_GROUP, clusters:
new ips(1) service: RPC_GROUP@@cbsp-limt1 -> [{"instanceId":"10.111.0.195#20000#null#cbsp-limt1","ip":"10.111.0.195","port":20000,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"RPC_GROUP@@cbsp-limt1","metadata":{"dubbo.metadata-service.url-params":"{\"prefer.serialization\":\"hessian2,fastjson2\",\"version\":\"2.0.0\",\"dubbo\":\"2.0.2\",\"release\":\"3.3.2\",\"side\":\"provider\",\"port\":\"20000\",\"protocol\":\"tri\"}","dubbo.endpoints":"[{\"port\":20000,\"protocol\":\"tri\"}]","dubbo.metadata.revision":"c88088292faa51b8a2f2b0d9dfa6250b","dubbo.metadata.storage-type":"local","meta-v":"2.0.0","timestamp":"1778636485744"},"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000,"ipDeleteTimeout":30000}]
```
QOS output when issue happens:
```text
As Consumer side:
+---------------------------------------------------------+----------+
| Consumer Service Name | NUM |
+---------------------------------------------------------+----------+
| cbsp-bp01/com.szfesc.cbsp.bp.api.BpServiceJsonApi |nacos-A(4)|
+---------------------------------------------------------+----------+
| cbsp-bp02/com.szfesc.cbsp.bp.api.BpServiceJsonApi |nacos-A(4)|
+---------------------------------------------------------+----------+
| cbsp-bpclt01/com.szfesc.cbsp.bpclt.api.BpJsonCtrlApi |nacos-A(2)|
+---------------------------------------------------------+----------+
| cbsp-bpclt02/com.szfesc.cbsp.bpclt.api.BpJsonCtrlApi |nacos-A(2)|
+---------------------------------------------------------+----------+
|cbsp-limt1/com.szfesc.cbsp.limt.api.LimitReProcServiceApi|nacos-A(0)|
+---------------------------------------------------------+----------+
|cbsp-limt2/com.szfesc.cbsp.limt.api.LimitReProcServiceApi|nacos-A(2)|
+---------------------------------------------------------+----------+
```
### Current suspicion
This may be related to the recovery path of application-level service discovery when mapping is initially empty.
Suspicious code areas:
- `ServiceDiscoveryRegistry.doSubscribe()`
- `ServiceDiscoveryRegistry.DefaultMappingListener.onEvent()`
- `ServiceDiscoveryRegistry.subscribeURLs()`
- `ServiceInstancesChangedListener.addListenerAndNotify()`
- `ServiceInstancesChangedListener.notifyAddressChanged()`
- `ServiceNameMapping.buildMappingKey()`
- `NacosMetadataReport.getServiceAppMapping()`
### Workaround
Changing:
```properties
dubbo.application.service-discovery.migration=FORCE_APPLICATION
```
to:
```properties
dubbo.application.service-discovery.migration=FORCE_INTERFACE
```
can avoid this issue, but this is only a workaround.
### Frequency
This issue is not 100% reproducible on every startup. It appears under certain startup timing conditions, especially when mapping is initially empty and later recovered by callback.
If needed, I can provide more logs and help test a patch.
### Do you have a (mini) reproduction demo?
- [ ] Yes, I have a minimal reproduction demo to help resolve this issue more effectively!
### Are you willing to submit a pull request to fix on your own?
- [x] Yes I am willing to submit a pull request on my own!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by reproducing the mapping-miss startup path with FORCE_APPLICATION and the two same-interface, different-group references. Read ServiceDiscoveryRegistry.doSubscribe(), DefaultMappingListener.onEvent(), subscribeURLs(), the ServiceInstancesChangedListener methods, ServiceNameMapping.buildMappingKey(), and NacosMetadataReport.getServiceAppMapping(). Done means both references complete application-level subscription, emit final address notifications, and show non-zero nacos-A counts in QoS output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100