apache / apache/incubator-seata
Seata-server 报错 ,导致seata-client无法注册。 Failed to submit a listener notification task. Event loop shut down
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 26k
- Forks
- 8.8k
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 4
Description
- [ ] I have searched the [issues](https://github.com/seata/seata/issues) of this repository and believe that this is not a duplicate.
### Ⅰ. Issue Description
表象:seata-server 循环报错,导致seata-client 注册超时。
### Ⅱ. Describe what happened
``` log
0x58a19b94, L:/10.20.9.3:8091 ! R:/10.20.13.181:49502], resourceSets=[]}
2020-05-04 09:37:45.388 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:338 -remove unused channel:[id: 0x75797452, L:/10.20.9.3:8091 ! R:/10.20.4.74:35468]
2020-05-04 09:37:45.389 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.11.169:33614 to server channel inactive.
2020-05-04 09:37:45.389 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:338 -remove unused channel:[id: 0x32e0f99a, L:/10.20.9.3:8091 ! R:/10.20.11.169:33614]
2020-05-04 09:37:45.389 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.11.169:33977 to server channel inactive.
2020-05-04 09:37:45.389 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:338 -remove unused channel:[id: 0xe77a16d4, L:/10.20.9.3:8091 ! R:/10.20.11.169:33977]
2020-05-04 09:37:45.390 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.4.8:53154 to server channel inactive.
2020-05-04 09:37:45.390 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:338 -remove unused channel:[id: 0x46a60285, L:/10.20.9.3:8091 ! R:/10.20.4.8:53154]
2020-05-04 09:37:45.390 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.11.169:34785 to server channel inactive.
2020-05-04 09:37:45.390 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:338 -remove unused channel:[id: 0x62771a9a, L:/10.20.9.3:8091 ! R:/10.20.11.169:34785]
2020-05-04 09:37:45.390 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.4.8:57160 to server channel inactive.
2020-05-04 09:37:45.390 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:338 -remove unused channel:[id: 0x01ad46e2, L:/10.20.9.3:8091 ! R:/10.20.4.8:57160]
2020-05-04 09:37:45.390 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.11.169:56142 to server channel inactive.
2020-05-04 09:37:45.390 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:334 -remove channel:[id: 0x03be50f6, L:/10.20.9.3:8091 ! R:/10.20.11.169:56142]context:RpcContext{applicationId='cas-store', transactionServiceGroup='cas_seata', clientId='cas-store:10.20.11.169:56142', channel=[id: 0x03be50f6, L:/10.20.9.3:8091 ! R:/10.20.11.169:56142], resourceSets=[]}
2020-05-04 09:37:45.390 WARN [NettyServerNIOWorker_1_15]io.netty.channel.nio.NioEventLoop.handleLoopException:483 -Unexpected exception in the selector loop.
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:124)
at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:107)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:120)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2020-05-04 09:37:45.397 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:334 -remove channel:[id: 0x060587be, L:/10.20.9.3:8091 ! R:/10.20.13.181:53198]context:RpcContext{applicationId='cas-systemadmin', transactionServiceGroup='cas_seata', clientId='cas-systemadmin:10.20.13.181:53198', channel=[id: 0x060587be, L:/10.20.9.3:8091 ! R:/10.20.13.181:53198], resourceSets=[]}
2020-05-04 09:37:45.397 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.4.74:43130 to server channel inactive.
2020-05-04 09:37:45.397 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:334 -remove channel:[id: 0xd829d74f, L:/10.20.9.3:8091 ! R:/10.20.4.74:43130]context:RpcContext{applicationId='cas-store', transactionServiceGroup='cas_seata', clientId='cas-store:10.20.4.74:43130', channel=[id: 0xd829d74f, L:/10.20.9.3:8091 ! R:/10.20.4.74:43130], resourceSets=[]}
2020-05-04 09:37:45.397 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.11.169:35720 to server channel inactive.
2020-05-04 09:37:45.397 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:334 -remove channel:[id: 0xcec3296d, L:/10.20.9.3:8091 ! R:/10.20.11.169:35720]context:RpcContext{applicationId='cas-store', transactionServiceGroup='cas_seata', clientId='cas-store:10.20.11.169:35720', channel=[id: 0xcec3296d, L:/10.20.9.3:8091 ! R:/10.20.11.169:35720], resourceSets=[]}
2020-05-04 09:37:46.387 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.11.169:56737 to server channel inactive.
2020-05-04 09:37:46.388 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:334 -remove channel:[id: 0x16647b0c, L:/10.20.9.3:8091 ! R:/10.20.11.169:56737]context:RpcContext{applicationId='cas-store', transactionServiceGroup='cas_seata', clientId='cas-store:10.20.11.169:56737', channel=[id: 0x16647b0c, L:/10.20.9.3:8091 ! R:/10.20.11.169:56737], resourceSets=null}
2020-05-04 09:37:46.390 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.DefaultServerMessageListenerImpl.onRegTmMessage:140 -checkAuth for client:10.20.11.169:35574,vgroup:cas_seata,applicationId:cas-store
2020-05-04 09:37:46.391 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.DefaultServerMessageListenerImpl.onRegTmMessage:140 -checkAuth for client:10.20.12.105:51142,vgroup:cas_seata,applicationId:cas-device
2020-05-04 09:37:46.391 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.11.169:56241 to server channel inactive.
2020-05-04 09:37:46.391 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:334 -remove channel:[id: 0xf034c9dd, L:/10.20.9.3:8091 ! R:/10.20.11.169:56241]context:RpcContext{applicationId='cas-store', transactionServiceGroup='cas_seata', clientId='cas-store:10.20.11.169:56241', channel=[id: 0xf034c9dd, L:/10.20.9.3:8091 ! R:/10.20.11.169:56241], resourceSets=null}
2020-05-04 09:37:46.391 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.11.169:35574 to server channel inactive.
2020-05-04 09:37:46.391 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:334 -remove channel:[id: 0x26d2ff2c, L:0.0.0.0/0.0.0.0:8091 ! R:/10.20.11.169:35574]context:RpcContext{applicationId='cas-store', transactionServiceGroup='cas_seata', clientId='cas-store:10.20.11.169:35574', channel=[id: 0x26d2ff2c, L:0.0.0.0/0.0.0.0:8091 ! R:/10.20.11.169:35574], resourceSets=null}
2020-05-04 09:37:46.391 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:329 -10.20.12.105:51142 to server channel inactive.
2020-05-04 09:37:46.391 INFO [NettyServerNIOWorker_1_15]io.seata.core.rpc.netty.RpcServer.handleDisconnect:334 -remove channel:[id: 0xcafe6e54, L:0.0.0.0/0.0.0.0:8091 ! R:/10.20.12.105:51142]context:RpcContext{applicationId='cas-device', transactionServiceGroup='cas_seata', clientId='cas-device:10.20.12.105:51142', channel=[id: 0xcafe6e54, L:0.0.0.0/0.0.0.0:8091 ! R:/10.20.12.105:51142], resourceSets=null}
2020-05-04 14:07:06.547 INFO [UndoLogDelete_1]io.seata.core.rpc.ChannelManager.tryOtherApp:467 -Removed inactive [id: 0x7bd1ebcc, L:0.0.0.0/0.0.0.0:8091 ! R:/10.20.13.181:40106]
2020-05-04 14:07:06.547 INFO [UndoLogDelete_1]io.seata.core.rpc.ChannelManager.tryOtherApp:467 -Removed inactive [id: 0xd7f03adb, L:/10.20.9.3:8091 ! R:/10.20.13.181:39952]
2020-05-04 14:07:06.548 ERROR[UndoLogDelete_1]i.netty.util.concurrent.DefaultPromise.rejectedExecution.safeExecute:770 -Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:855)
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:328)
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:321)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:778)
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:768)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:432)
at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:112)
at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:89)
at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1010)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:825)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831)
at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071)
at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300)
at io.seata.core.rpc.netty.AbstractRpcRemoting.sendSingleRequest(AbstractRpcRemoting.java:282)
at io.seata.core.rpc.netty.AbstractRpcRemoting.sendAsyncRequest(AbstractRpcRemoting.java:261)
at io.seata.core.rpc.netty.AbstractRpcRemoting.sendAsyncRequestWithoutResponse(AbstractRpcRemoting.java:210)
at io.seata.core.rpc.netty.RpcServer.sendASyncRequest(RpcServer.java:279)
at io.seata.server.coordinator.DefaultCoordinator.undoLogDelete(DefaultCoordinator.java:484)
at io.seata.server.coordinator.DefaultCoordinator.lambda$init$5(DefaultCoordinator.java:529)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
```
### Ⅲ. Describe what you expected to happen
seata-client 正常注册,seata-server 正常。
### Ⅳ. How to reproduce it (as minimally and precisely as possible)
不清楚。服务seata-server 突然就这样了,问题爆发是因为seata-client 无法注册。
### Ⅴ. Anything else we need to know?
### Ⅵ. Environment:
- JDK version :1.8
- OS : CentOS Linux release 7.7.1908
- Others:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with io.seata.core.rpc.netty.RpcServer.handleDisconnect and the Netty event-loop stack trace, then trace io.seata.server.coordinator.DefaultCoordinator.undoLogDelete and AbstractRpcRemoting.sendAsyncRequest. Reproduce the registration failure with the reported JDK and CentOS environment if possible; done means the event loop remains usable, the error no longer recurs, and seata-client registration succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100