apache / apache/incubator-seata

channel:null CPU 100% Lock global session failed

Open
#3,294 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
26k
Forks
8.8k
Avg merge
1d 8h
Merged PRs (30d)
4

Description

- [x] I have searched the [issues](https://github.com/seata/seata/issues) of this repository and believe that this is not a duplicate.

CPU 负荷100%,
查看报错日志如下,看样子像是空指针引发的,不太确定,数据库也100%,不知道是seata影响的数据库,还是数据库CPU满了,影响的seata.希望能帮忙解答是什么原因引起的,怎么能解决这个问题.谢谢

12:58:22,939 INFO Register branch successfully, xid = 172.121.33.2:1091:2028855554, branchId = 2028855558, resourceId = jdbc:mysql://*********s ,lockKeys = a_credits:478499968109502464

12:58:23,479 INFO channel:[id: 0x975c23f6, L:/172.121.33.2:1091 - R:/172.17.224.46:40844] read idle.

12:58:23,479 INFO Begin new global transaction applicationId: tiyu-app,transactionServiceGroup: tiyu-app-group, transactionName: fulfilDailyTask(com.tiyu.common.basic.bean.ResponseBean, java.lang.String, java.lang.String),timeout:60000,xid:172.121.33.2:1091:2028855560

12:58:23,479 INFO SeataMergeMessage timeout=60000,transactionName=fulfilDailyTask(com.tiyu.common.basic.bean.ResponseBean, java.lang.String, java.lang.String)
,clientIp:172.17.224.47,vgroup:tiyu-app-group

12:58:24,001 INFO 172.17.224.46:40844 to server channel inactive.

12:58:24,001 INFO remove channel:[id: 0x975c23f6, L:/172.121.33.2:1091 - R:/172.17.224.46:40844]context:RpcContext{applicationId='tiyu-doctor', transactionServiceGroup='tiyu-doctor-group', clientId='tiyu-doctor:172.17.224.46:40844', channel=[id: 0x975c23f6, L:/172.121.33.2:1091 - R:/172.17.224.46:40844], resourceSets=null}

12:58:24,001 INFO closeChannelHandlerContext channel:[id: 0x975c23f6, L:/172.121.33.2:1091 - R:/172.17.224.46:40844]

12:58:24,524 INFO SeataMergeMessage xid=172.121.33.2:1091:2028855551,branchId=2028855555,resourceId=null,status=PhaseOne_Done,applicationData=null
,clientIp:172.17.224.109,vgroup:tiyu-credits-group

12:58:24,524 INFO Report branch status successfully, xid = 172.121.33.2:1091:2028855553, branchId = 2028855559

12:58:24,524 INFO 172.17.224.46:40844 to server channel inactive.

12:58:25,038 INFO SeataMergeMessage xid=172.121.33.2:1091:2028855554,branchType=AT,resourceId=jdbc:mysql://*********s,lockKey=a_credits:478499968109502464
,clientIp:172.17.224.58,vgroup:tiyu-credits-group

12:58:30,495 INFO SeataMergeMessage xid=172.121.33.2:1091:2028855556,branchId=2028855563,resourceId=null,status=PhaseOne_Done,applicationData=null
xid=172.121.33.2:1091:2028855560,branchId=2028855564,resourceId=null,status=PhaseOne_Done,applicationData=null
,clientIp:172.17.224.109,vgroup:tiyu-credits-group

12:58:31,017 INFO 172.17.81.93:51532 to server channel inactive.

12:58:31,522 INFO Register branch successfully, xid = 172.121.33.2:1091:2028855567, branchId = 2028855571, resourceId = jdbc:mysql://*********s ,lockKeys = a_credits:486043549581959168

12:58:31,017 INFO SeataMergeMessage timeout=60000,transactionName=fulfilDailyTask(com.tiyu.common.basic.bean.ResponseBean, java.lang.String, java.lang.String)
,clientIp:172.17.224.49,vgroup:tiyu-app-group

12:58:32,578 INFO SeataMergeMessage xid=172.121.33.2:1091:2028855551,branchId=2028855561,resourceId=null,status=PhaseOne_Done,applicationData=null
,clientIp:172.17.224.58,vgroup:tiyu-credits-group

12:58:32,578 INFO Begin new global transaction applicationId: tiyu-app,transactionServiceGroup: tiyu-app-group, transactionName: fulfilDailyTask(com.tiyu.common.basic.bean.ResponseBean, java.lang.String, java.lang.String),timeout:60000,xid:172.121.33.2:1091:2028855573

12:58:32,578 INFO channel:[id: 0x57f9bb72, L:/172.121.33.2:1091 - R:/172.17.81.93:51586] read idle.

12:58:32,578 INFO channel:[id: 0xfe4300d3, L:/172.121.33.2:1091 - R:/172.17.224.46:41168] read idle.

12:58:32,034 INFO Report branch status successfully, xid = 172.121.33.2:1091:2028855554, branchId = 2028855568

12:58:32,034 INFO Begin new global transaction applicationId: tiyu-app,transactionServiceGroup: tiyu-app-group, transactionName: fulfilDailyTask(com.tiyu.common.basic.bean.ResponseBean, java.lang.String, java.lang.String),timeout:60000,xid:172.121.33.2:1091:2028855572

12:58:33,120 INFO Begin new global transaction applicationId: tiyu-app,transactionServiceGroup: tiyu-app-group, transactionName: fulfilDailyTask(com.tiyu.common.basic.bean.ResponseBean, java.lang.String, java.lang.String),timeout:60000,xid:172.121.33.2:1091:2028855574

12:58:33,120 INFO 172.17.81.93:51586 to server channel inactive.

12:58:34,729 INFO 172.17.224.47:16398 to server channel inactive.

12:58:34,729 INFO closeChannelHandlerContext channel:[id: 0x09d72b80, L:/172.121.33.2:1091 - R:/172.17.224.47:16382]

12:58:34,730 INFO RM register success,message:RegisterRMRequest{resourceIds='null', applicationId='tiyu-doctor', transactionServiceGroup='tiyu-doctor-group'},channel:[id: 0xadee59f4, L:/172.121.33.2:1091 - R:/172.17.81.93:51748]

12:58:34,730 INFO closeChannelHandlerContext channel:[id: 0x57f9bb72, L:/172.121.33.2:1091 - R:/172.17.81.93:51586]

12:58:41,540 INFO Exception timeout checking ...

io.seata.core.exception.GlobalTransactionException: Lock global session failed
at io.seata.server.session.GlobalSession$GlobalSessionLock.lock(GlobalSession.java:624)
at io.seata.server.session.GlobalSession.lock(GlobalSession.java:603)
at io.seata.server.storage.file.session.FileSessionManager.lockAndExecute(FileSessionManager.java:151)
at io.seata.server.session.SessionHolder.lockAndExecute(SessionHolder.java:266)
at io.seata.server.coordinator.DefaultCoordinator.timeoutCheck(DefaultCoordinator.java:229)
at io.seata.server.coordinator.DefaultCoordinator.lambda$init$4(DefaultCoordinator.java:404)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
12:58:45,509 INFO remove channel:[id: 0xf1662c43, L:/172.121.33.2:1091 ! R:/172.17.224.49:30610]context:RpcContext{applicationId='tiyu-app', transactionServiceGroup='tiyu-app-group', clientId='tiyu-app:172.17.224.49:30610', channel=[id: 0xf1662c43, L:/172.121.33.2:1091 ! R:/172.17.224.49:30610], resourceSets=null}

12:58:46,030 INFO Register branch successfully, xid = 172.121.33.2:1091:2028855580, branchId = 2028855586, resourceId = jdbc:mysql://*********s ,lockKeys = a_credits:430559922115305472

12:58:47,095 INFO channel:[id: 0xa4e24a78, L:/172.121.33.2:1091 - R:/172.17.224.49:18472] read idle.

12:58:46,567 INFO SeataMergeMessage xid=172.121.33.2:1091:2028855553,extraData=null
,clientIp:172.17.224.108,vgroup:tiyu-app-group

12:59:46,930 INFO 172.17.224.58:33338 to server channel inactive.

13:00:21,591 ERROR 0104

java.lang.RuntimeException: channel is error. channel:null
at io.seata.core.rpc.netty.RpcServer.sendResponse(RpcServer.java:92)
at io.seata.core.rpc.DefaultServerMessageListenerImpl.onTrxMessage(DefaultServerMessageListenerImpl.java:95)
at io.seata.core.rpc.netty.AbstractRpcRemotingServer$ServerHandler.dispatch(AbstractRpcRemotingServer.java:196)
at io.seata.core.rpc.netty.AbstractRpcRemoting$AbstractHandler$1.run(AbstractRpcRemoting.java:452)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
13:00:20,377 INFO 172.17.224.47:16452 to server channel inactive.

13:00:16,167 INFO 172.17.224.47:48904 to server channel inactive.

13:00:01,734 INFO 172.17.81.94:51550 to server channel inactive.

13:00:03,805 INFO 172.17.224.109:42526 to server channel inactive.

13:00:16,167 ERROR wait response error:cost 63384 ms,ip:null,request:xid=172.121.33.2:1091:2028855554,branchId=2028855558,branchType=AT,resourceId=jdbc:mysql://*********s,applicationData=null

13:00:08,069 INFO 172.17.224.58:33336 to server channel inactive.

13:01:49,322 ERROR 0104

java.lang.NullPointerException: null
at io.seata.core.rpc.ChannelManager.getSameClientChannel(ChannelManager.java:245)
at io.seata.core.rpc.netty.RpcServer.sendResponse(RpcServer.java:87)
at io.seata.core.rpc.DefaultServerMessageListenerImpl.onRegRmMessage(DefaultServerMessageListenerImpl.java:125)
at io.seata.core.rpc.netty.AbstractRpcRemotingServer$ServerHandler.dispatch(AbstractRpcRemotingServer.java:193)
at io.seata.core.rpc.netty.AbstractRpcRemoting$AbstractHandler$1.run(AbstractRpcRemoting.java:452)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
13:01:47,494 ERROR 0104

java.lang.NullPointerException: null
at io.seata.core.rpc.ChannelManager.getSameClientChannel(ChannelManager.java:245)
at io.seata.core.rpc.netty.RpcServer.sendResponse(RpcServer.java:87)
at io.seata.core.rpc.DefaultServerMessageListenerImpl.onRegRmMessage(DefaultServerMessageListenerImpl.java:125)
at io.seata.core.rpc.netty.AbstractRpcRemotingServer$ServerHandler.dispatch(AbstractRpcRemotingServer.java:193)
at io.seata.core.rpc.netty.AbstractRpcRemoting$AbstractHandler$1.run(AbstractRpcRemoting.java:452)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

Contributor guide

Open the contributing guide

Research direction

Start with the stack traces in ChannelManager.java:245, RpcServer.java:87-92, GlobalSession.java:624, and FileSessionManager.java:151, then trace the channel and global-session locking paths. Reproduce the CPU and database load if possible and determine whether the null channel or lock failure is causal; done means a confirmed cause, a bounded fix, and regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.