apache / apache/dubbo

[Bug] No provider available in test environment when starting dubbo service in local environment with a different group setting

Open
#14,562 2 comments 0 reactions 0 assignees View on GitHub
component/need-triage type/need-triage
Dominant language
Java
Stars
41.6k
Forks
26.4k
Avg merge
15h 13m
Merged PRs (30d)
4

Description

### Pre-check

- [X] I am sure that all the content I provide is in English.

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.

### Apache Dubbo Component

Java SDK (apache/dubbo)

### Dubbo Version

dubbo 3.2.13, openjdk 1.8

### Steps to reproduce this issue

To facilitate local debugging, we typically use data and invoke Dubbo services from the test environment without affecting service interactions within that environment. We run our application in the local development environment with the same namespace as the test environment but with a different Dubbo group (e.g., the group is test in the test environment and local in the local environment).

This setup worked well across many versions until we upgraded to 3.2.13. Now, when we start the application locally, it affects the test environment, the test and local environment has the same services in differrent group, and the invocation in the test environment will have the exception like the followings. The exception observed in the test environment is as follows. Using different namespaces avoids this issue, but then we can't access the data and providers in the test environment.

`
2024-08-20 21:33:02.698|ERROR |197978|http-nio-19405-exec-6|c.c.d.s.c.a.ExceptionHandlerAdvice.(:)|[TID: N/A]|RpcException exception is
error message:Failed to invoke the method query in the service com.example.QueryService. No provider available for the service TEST/com.example.QueryService:1.0.0 from registry ServiceDiscoveryRegistryDirectory(registry: 10.0.1.29:1811, subscribed key: [query-dubbo])-Directory(invokers: 1[10.0.1.29:1940], validInvokers: 1[10.0.1.29:1940], invokersToReconnect: 0[]) on the consumer 10.0.1.29 using the dubbo version 3.2.13. Please check if the providers have been started and registered.
org.apache.dubbo.rpc.RpcException: Failed to invoke the method query in the service com.example.QueryService. No provider available for the service TEST/com.example.QueryService:1.0.0 from registry ServiceDiscoveryRegistryDirectory(registry: 10.0.1.29:1811, subscribed key: [query-dubbo])-Directory(invokers: 1[10.0.1.29:1940], validInvokers: 1[10.0.1.29:1940], invokersToReconnect: 0[]) on the consumer 10.0.1.29 using the dubbo version 3.2.13. Please check if the providers have been started and registered.
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.checkInvokers(AbstractClusterInvoker.java:397)
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:358)
at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:108)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:57)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.support.ObservationSenderFilter.invoke(ObservationSenderFilter.java:62)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:40)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.spring.security.filter.ContextHolderAuthenticationPrepareFilter.invoke(ContextHolderAuthenticationPrepareFilter.java:68)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:86)
at org.apache.dubbo.rpc.cluster.filter.support.MetricsConsumerFilter.invoke(MetricsConsumerFilter.java:38)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.support.ConsumerContextFilter.invoke(ConsumerContextFilter.java:119)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197)
at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$ClusterFilterInvoker.invoke(AbstractCluster.java:101)
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:106)
at org.apache.dubbo.rpc.cluster.support.wrapper.ScopeClusterInvoker.invoke(ScopeClusterInvoker.java:171)
at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:294)
at org.apache.dubbo.rpc.proxy.InvocationUtil.invoke(InvocationUtil.java:64)
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:81)
`

`
2024-08-20 21:33:02.698|ERROR |197978|http-nio-19405-exec-3|o.a.d.r.c.SingleRouterChain.(:)|[TID: N/A]| [DUBBO] Reject to route, because the invokers has changed., dubbo version: 3.2.13, current host: 10.0.1.29, error code: 99-0. This may be caused by , go to https://dubbo.apache.org/faq/99/0 to find instructions. Router's invoker size: 1 Invocation's invoker size: 1
2024-08-20 21:33:02.698|ERROR |197978|http-nio-19405-exec-3|o.a.d.r.i.DynamicDirectory.(:)|[TID: N/A]| [DUBBO] Failed to execute router: nacos://10.0.1.29:1811/org.apache.dubbo.registry.RegistryService?REGISTRY_CLUSTER=default:PROD&application=demo-dubbo&check.serializable=false&dubbo=2.0.2&executor-management-mode=isolation&file=/data/tmp/.dubbo/dubbo-registry-dubbo-consumer-demo-0.cache&file-cache=true&namespace=PROD&password=Nacos_654321&pid=197978&qos.enable=false&release=3.2.13&serialize.check.status=DISABLE&simplified=true×tamp=1723729304540&username=nacos, cause: reject to route, because the invokers has changed., dubbo version: 3.2.13, current host: 10.0.1.29, error code: 2-1. This may be caused by , go to https://dubbo.apache.org/faq/2/1 to find instructions.
java.lang.IllegalStateException: reject to route, because the invokers has changed.
at org.apache.dubbo.rpc.cluster.SingleRouterChain.route(SingleRouterChain.java:147) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.registry.integration.DynamicDirectory.doList(DynamicDirectory.java:214) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.directory.AbstractDirectory.list(AbstractDirectory.java:232) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.list(AbstractClusterInvoker.java:452) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:355) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:108) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:57) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.support.ObservationSenderFilter.invoke(ObservationSenderFilter.java:62) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:40) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.spring.security.filter.ContextHolderAuthenticationPrepareFilter.invoke(ContextHolderAuthenticationPrepareFilter.java:68) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:86) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.support.MetricsConsumerFilter.invoke(MetricsConsumerFilter.java:38) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.support.ConsumerContextFilter.invoke(ConsumerContextFilter.java:119) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$ClusterFilterInvoker.invoke(AbstractCluster.java:101) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:106) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.cluster.support.wrapper.ScopeClusterInvoker.invoke(ScopeClusterInvoker.java:171) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:294) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.proxy.InvocationUtil.invoke(InvocationUtil.java:64) ~[dubbo-3.2.13.jar:3.2.13]
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:81) ~[dubbo-3.2.13.jar:3.2.13]
`

### What you expected to happen

I expect to be able to start the application with Dubbo services in the local environment, allowing it to invoke providers in the test environment without affecting the test environment itself.

### Anything else

_No response_

### Are you willing to submit a pull request to fix on your own?

- [x] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Reproduce with Dubbo 3.2.13 using the same namespace and different groups, then inspect the stack-trace entry points in SingleRouterChain.java, DynamicDirectory.java, and AbstractClusterInvoker.java. Compare provider routing and registry changes when the local application starts. Done means test-environment invocations remain available without the reported no-provider or invoker-change errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.