apache / apache/dubbo

Dubbo3.0.2.1 Failed to invoke remote method: getMetadataInfo, provider:

Open
#8,791 9 comments 1 reaction 0 assignees View on GitHub
help wanted
Dominant language
Java
Stars
41.6k
Forks
26.4k
Avg merge
15h 13m
Merged PRs (30d)
4

Description

- [ ] I have searched the [issues](https://github.com/apache/dubbo/issues) of this repository and believe that this is not a duplicate.
- [ ] I have checked the [FAQ](https://github.com/apache/dubbo/blob/master/FAQ.md) of this repository and believe that this is not a duplicate.

### Environment

* Dubbo version: 3.0.2.1
* Operating System version: Centos7
* Java version: 1.8
[dubbo错误信息.txt](https://github.com/apache/dubbo/files/7158383/dubbo.txt)

### Steps to reproduce this issue

1. A、B Service Cross reference

### Expected Result

Fallback Dubbo version 3.0.1 does not have this problem

### Actual Result

What actually happens?

After I update Dubbo version 3.0.1 to 3.0.2.1, the request service result returns to normal, but the exception "failed to invoke remote method: getmetadatainfo, provider:" will be thrown

```
HTTP Status 400 – Bad Requestbody {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}

HTTP Status 400 – Bad Request

in channel: NettyChannel [channel=[id: 0xc33d6db0, L:/192.168.31.105:61470 - R:/192.168.31.105:9211]], url: dubbo://192.168.31.105:9211/org.apache.dubbo.metadata.MetadataService?codec=dubbo&getAndListenInstanceMetadata.1.callback=true&group=api-auth&heartbeat=60000&reconnect=false&side=consumer&timeout=5000&version=1.0.0, dubbo version: 3.0.2.1, current host: 192.168.31.105
java.lang.Exception: Dubbo client can not supported string message: HTTP/1.1 400
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 435
Date: Mon, 13 Sep 2021 05:36:36 GMT
Connection: close

HTTP Status 400 – Bad Requestbody {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}

HTTP Status 400 – Bad Request

in channel: NettyChannel [channel=[id: 0xc33d6db0, L:/192.168.31.105:61470 - R:/192.168.31.105:9211]], url: dubbo://192.168.31.105:9211/org.apache.dubbo.metadata.MetadataService?codec=dubbo&getAndListenInstanceMetadata.1.callback=true&group=api-auth&heartbeat=60000&reconnect=false&side=consumer&timeout=5000&version=1.0.0
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:184)
at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
at java.lang.Thread.run(Thread.java:748)
13:36:36.259 [nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent] ERROR o.a.d.r.c.e.l.ServiceInstancesChangedListener - [doGetMetadataInfo,341] - [DUBBO] Failed to load service metadata, meta type is local, dubbo version: 3.0.2.1, current host: 192.168.31.105
org.apache.dubbo.rpc.RpcException: Failed to invoke remote method: getMetadataInfo, provider: dubbo://192.168.31.105:9211/org.apache.dubbo.metadata.MetadataService?getAndListenInstanceMetadata.1.callback=true&group=api-auth&reconnect=false&side=consumer&timeout=5000&version=1.0.0, cause: org.apache.dubbo.remoting.RemotingException: Channel NettyChannel [channel=[id: 0xc33d6db0, L:/192.168.31.105:61470 ! R:/192.168.31.105:9211]] is inactive. Directly return the unFinished request : Request [id=1, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=getMetadataInfo, parameterTypes=[class java.lang.String], arguments=[null], attachments={group=api-auth, path=org.apache.dubbo.metadata.MetadataService, version=1.0.0}]]
at org.apache.dubbo.rpc.protocol.AbstractInvoker.waitForResultIfSync(AbstractInvoker.java:283)
at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:181)
at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78)
at io.seata.integration.dubbo.ApacheDubboTransactionPropagationFilter.invoke(ApacheDubboTransactionPropagationFilter.java:69)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$FilterChainNode.invoke(FilterChainBuilder.java:82)
at com.alibaba.dubbo.rpc.Invoker$CompatibleInvoker.invoke(Invoker.java:55)
at io.seata.integration.dubbo.alibaba.AlibabaDubboTransactionPropagationFilter.invoke(AlibabaDubboTransactionPropagationFilter.java:45)
at com.alibaba.dubbo.rpc.Filter.invoke(Filter.java:29)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$FilterChainNode.invoke(FilterChainBuilder.java:82)
at com.alibaba.csp.sentinel.adapter.dubbo.SentinelDubboConsumerFilter.syncInvoke(SentinelDubboConsumerFilter.java:82)
at com.alibaba.csp.sentinel.adapter.dubbo.SentinelDubboConsumerFilter.invoke(SentinelDubboConsumerFilter.java:66)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$FilterChainNode.invoke(FilterChainBuilder.java:82)
at com.alibaba.csp.sentinel.adapter.dubbo.DubboAppContextFilter.invoke(DubboAppContextFilter.java:43)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$FilterChainNode.invoke(FilterChainBuilder.java:82)
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:95)
at org.apache.dubbo.common.bytecode.proxy2.getMetadataInfo(proxy2.java)
at org.apache.dubbo.registry.client.event.listener.ServiceInstancesChangedListener.doGetMetadataInfo(ServiceInstancesChangedListener.java:338)
at org.apache.dubbo.registry.client.event.listener.ServiceInstancesChangedListener.getRemoteMetadata(ServiceInstancesChangedListener.java:291)
at org.apache.dubbo.registry.client.event.listener.ServiceInstancesChangedListener.onEvent(ServiceInstancesChangedListener.java:138)
at org.apache.dubbo.registry.nacos.NacosServiceDiscovery.handleEvent(NacosServiceDiscovery.java:149)
at org.apache.dubbo.registry.nacos.NacosServiceDiscovery.lambda$addServiceInstancesChangedListener$4(NacosServiceDiscovery.java:128)
at com.alibaba.nacos.client.naming.event.InstancesChangeNotifier.onEvent(InstancesChangeNotifier.java:124)
at com.alibaba.nacos.client.naming.event.InstancesChangeNotifier.onEvent(InstancesChangeNotifier.java:40)
at com.alibaba.nacos.common.notify.DefaultPublisher.lambda$notifySubscriber$0(DefaultPublisher.java:201)
at com.alibaba.nacos.common.notify.DefaultPublisher.notifySubscriber(DefaultPublisher.java:208)
at com.alibaba.nacos.common.notify.DefaultPublisher.receiveEvent(DefaultPublisher.java:192)
at com.alibaba.nacos.common.notify.DefaultPublisher.openEventHandler(DefaultPublisher.java:118)
at com.alibaba.nacos.common.notify.DefaultPublisher.run(DefaultPublisher.java:95)
Caused by: java.util.concurrent.ExecutionException: org.apache.dubbo.remoting.RemotingException: Channel NettyChannel [channel=[id: 0xc33d6db0, L:/192.168.31.105:61470 ! R:/192.168.31.105:9211]] is inactive. Directly return the unFinished request : Request [id=1, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=getMetadataInfo, parameterTypes=[class java.lang.String], arguments=[null], attachments={group=api-auth, path=org.apache.dubbo.metadata.MetadataService, version=1.0.0}]]
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:181)
at org.apache.dubbo.rpc.protocol.AbstractInvoker.waitForResultIfSync(AbstractInvoker.java:272)
... 27 common frames omitted
Caused by: org.apache.dubbo.remoting.RemotingException: Channel NettyChannel [channel=[id: 0xc33d6db0, L:/192.168.31.105:61470 ! R:/192.168.31.105:9211]] is inactive. Directly return the unFinished request : Request [id=1, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=getMetadataInfo, parameterTypes=[class java.lang.String], arguments=[null], attachments={group=api-auth, path=org.apache.dubbo.metadata.MetadataService, version=1.0.0}]]
at org.apache.dubbo.remoting.exchange.support.DefaultFuture.doReceived(DefaultFuture.java:207)
at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:170)
at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:158)
at org.apache.dubbo.remoting.exchange.support.DefaultFuture.closeChannel(DefaultFuture.java:151)
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.disconnected(HeaderExchangeHandler.java:134)
at org.apache.dubbo.remoting.transport.AbstractChannelHandlerDelegate.disconnected(AbstractChannelHandlerDelegate.java:48)
at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:73)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
at java.lang.Thread.run(Thread.java:748)
13:36:36.261 [nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent] ERROR o.a.d.r.c.e.l.ServiceInstancesChangedListener - [getRemoteMetadata,297] - [DUBBO] Failed to get MetadataInfo for instance 192.168.31.105:9211?revision=null&cluster=null, wait for retry., dubbo version: 3.0.2.1, current host: 192.168.31.105
13:36:37.266 [DubboClientHandler-thread-1] ERROR o.a.d.r.e.s.h.HeaderExchangeHandler - [received,185] - [DUBBO] Dubbo client can not supported string message: HTTP/1.1 400
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 435
Date: Mon, 13 Sep 2021 05:36:37 GMT
Connection: close

```

Contributor guide

Open the contributing guide

Research direction

Start with ServiceInstancesChangedListener.java around doGetMetadataInfo and getRemoteMetadata, then trace the reported failure through HeaderExchangeHandler.java and the Netty channel stack. Reproduce the A/B service cross-reference with Dubbo 3.0.2.1 and compare it with 3.0.1; done means metadata retrieval no longer produces the HTTP 400 and failed getMetadataInfo errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.