alibaba / alibaba/otter

node日志报错,好像是超时

Open
#253 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.1k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Description

Caused by: com.alibaba.dubbo.rpc.RpcException: Invoke remote method timeout. method: acceptEvent, provider: dubbo://172.16.51.223:2088/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&iothreads=4&lazy=true&serialization=java&threads=50, cause: Waiting server-side response timeout by scan timer. start time: 2017-01-09 13:10:46.653, end time: 2017-01-09 13:11:36.659, client elapsed: 0 ms, server elapsed: 50006 ms, timeout: 50000 ms, request: Request [id=9143603, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=acceptEvent, parameterTypes=[class com.alibaba.otter.shared.communication.core.model.Event], arguments=[StageSingleEvent[pipelineId=15,stage=EXTRACT,data=EtlEventData[currNid=2,nextNid=6,desc=[HttpPipeKey[identity=Identity[channelId=9,pipelineId=15,processId=97514],crc=,key=,url=http://192.168.4.243:9090/download/DbBatch-2017-01-09-13-09-03-9-15-97514.gzip,dataType=DB_BATCH]],processId=97514,startTime=1483938539488,endTime=,firstTime=1483935245000,batchId=2909,number=29502,size=,exts=,pipelineId=15],type=stageSingle]], attachments={path=endpoint, version=0.0.0}]], channel: /192.168.4.243:42147 -> /172.16.51.223:2088
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:99) ~[dubbo-2.5.3.jar:2.5.3]
at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144) ~[dubbo-2.5.3.jar:2.5.3]
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52) ~[dubbo-2.5.3.jar:2.5.3]
at com.alibaba.dubbo.common.bytecode.proxy3.acceptEvent(proxy3.java) ~[na:2.5.3]
at com.alibaba.otter.shared.communication.core.impl.dubbo.DubboCommunicationConnection.call(DubboCommunicationConnection.java:45) ~[shared.communication-4.2.13.jar:na]
at com.alibaba.otter.shared.communication.core.impl.DefaultCommunicationClientImpl.call(DefaultCommunicationClientImpl.java:96) ~[shared.communication-4.2.13.jar:na]
at com.alibaba.otter.shared.arbitrate.impl.communication.ArbitrateCommmunicationClient.call(ArbitrateCommmunicationClient.java:52) ~[shared.arbitrate-4.2.13.jar:na]
at com.alibaba.otter.shared.arbitrate.impl.setl.rpc.RpcStageEventDispatcher.single(RpcStageEventDispatcher.java:69) ~[shared.arbitrate-4.2.13.jar:na]
at com.alibaba.otter.shared.arbitrate.impl.setl.rpc.ExtractRpcArbitrateEvent.single(ExtractRpcArbitrateEvent.java:79) ~[shared.arbitrate-4.2.13.jar:na]
at com.alibaba.otter.shared.arbitrate.impl.setl.delegate.ExtractDelegateArbitrateEvent.single(ExtractDelegateArbitrateEvent.java:40) ~[shared.arbitrate-4.2.13.jar:na]
at com.alibaba.otter.shared.arbitrate.impl.setl.delegate.ExtractDelegateArbitrateEvent$$FastClassByCGLIB$$b36a599b.invoke() ~[cglib-nodep-2.2.jar:na]
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) ~[cglib-nodep-2.2.jar:na]
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) ~[spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]
at com.alibaba.otter.shared.arbitrate.impl.interceptor.LogInterceptor.invoke(LogInterceptor.java:53) ~[shared.arbitrate-4.2.13.jar:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622) ~[spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]
at com.alibaba.otter.shared.arbitrate.impl.setl.delegate.ExtractDelegateArbitrateEvent$$EnhancerByCGLIB$$dcc882e7.single() ~[cglib-nodep-2.2.jar:na]
at com.alibaba.otter.node.etl.extract.ExtractTask$1.run(ExtractTask.java:98) ~[node.etl-4.2.13.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: com.alibaba.dubbo.remoting.TimeoutException: Waiting server-side response timeout by scan timer. start time: 2017-01-09 13:10:46.653, end time: 2017-01-09 13:11:36.659, client elapsed: 0 ms, server elapsed: 50006 ms, timeout: 50000 ms, request: Request [id=9143603, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=acceptEvent, parameterTypes=[class com.alibaba.otter.shared.communication.core.model.Event], arguments=[StageSingleEvent[pipelineId=15,stage=EXTRACT,data=EtlEventData[currNid=2,nextNid=6,desc=[HttpPipeKey[identity=Identity[channelId=9,pipelineId=15,processId=97514],crc=,key=,url=http://192.168.4.243:9090/download/DbBatch-2017-01-09-13-09-03-9-15-97514.gzip,dataType=DB_BATCH]],processId=97514,startTime=1483938539488,endTime=,firstTime=1483935245000,batchId=2909,number=29502,size=,exts=,pipelineId=15],type=stageSingle]], attachments={path=endpoint, version=0.0.0}]], channel: /192.168.4.243:42147 -> /172.16.51.223:2088
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.returnFromResponse(DefaultFuture.java:188) ~[dubbo-2.5.3.jar:2.5.3]
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:110) ~[dubbo-2.5.3.jar:2.5.3]
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:84) ~[dubbo-2.5.3.jar:2.5.3]
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:96) ~[dubbo-2.5.3.jar:2.5.3]
... 23 common frames omitted

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported stack-trace entry points in com.alibaba.otter.node.etl.extract.ExtractTask, ExtractRpcArbitrateEvent, RpcStageEventDispatcher, and DubboCommunicationConnection. Trace the acceptEvent call and the server-side 50,000 ms timeout; done requires identifying the cause and documenting or testing a verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.