apache / apache/incubator-seata
Cannot get a connection, pool error Timeout waiting for idle object
- Dominant language
- Java
- Stars
- 26k
- Forks
- 8.8k
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 4
Description
- [ ] I have searched the [issues](https://github.com/seata/seata/issues) of this repository and believe that this is not a duplicate.
### Ⅰ. Issue Description
线上环境,运行版本:1.0.0 ,这两天突然出io.seata.core.exception.TransactionException: branch register request failed. xid=xx.xx.xx.xx:8091:2336353978, msg=Cannot get a connection, pool error Timeout waiting for idle object
### Ⅱ. Describe what happened
线上环境凌晨会有批处理任务执行导致
If there is an exception, please attach the exception trace:
```
io.seata.core.exception.TransactionException: branch register request failed. xid=xxx.xxx.xxx.xxx:8091:2336353978, msg=Cannot get a connection, pool error Timeout waiting for idle object
at io.seata.server.AbstractTCInboundHandler$4.execute(AbstractTCInboundHandler.java:175)
at io.seata.server.AbstractTCInboundHandler$4.execute(AbstractTCInboundHandler.java:167)
at io.seata.core.exception.AbstractExceptionHandler.exceptionHandleTemplate(AbstractExceptionHandler.java:117)
at io.seata.server.AbstractTCInboundHandler.handle(AbstractTCInboundHandler.java:167)
at io.seata.core.protocol.transaction.BranchRegisterRequest.handle(BranchRegisterRequest.java:136)
at io.seata.server.coordinator.DefaultCoordinator.onRequest(DefaultCoordinator.java:544)
at io.seata.core.rpc.DefaultServerMessageListenerImpl.onTrxMessage(DefaultServerMessageListenerImpl.java:91)
at io.seata.core.rpc.netty.RpcServer.dispatch(RpcServer.java:295)
at io.seata.core.rpc.netty.AbstractRpcRemoting$3.run(AbstractRpcRemoting.java:384)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.seata.common.exception.StoreException: Cannot get a connection, pool error Timeout waiting for idle object
at io.seata.core.store.db.LockStoreDataBaseDAO.acquireLock(LockStoreDataBaseDAO.java:186)
at io.seata.server.lock.db.DataBaseLocker.acquireLock(DataBaseLocker.java:64)
at io.seata.server.lock.AbstractLockManager.acquireLock(AbstractLockManager.java:59)
at io.seata.server.session.BranchSession.lock(BranchSession.java:273)
at io.seata.server.coordinator.DefaultCore.lambda$branchRegister$0(DefaultCore.java:85)
at io.seata.server.session.GlobalSession.lockAndExcute(GlobalSession.java:605)
at io.seata.server.coordinator.DefaultCore.branchRegister(DefaultCore.java:70)
at io.seata.server.coordinator.DefaultCoordinator.doBranchRegister(DefaultCoordinator.java:203)
at io.seata.server.AbstractTCInboundHandler$4.execute(AbstractTCInboundHandler.java:172)
... 12 common frames omitted
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at io.seata.core.store.db.LockStoreDataBaseDAO.acquireLock(LockStoreDataBaseDAO.java:114)
... 20 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 22 common frames omitted
```
### Ⅲ. Describe what you expected to happen
### Ⅳ. How to reproduce it (as minimally and precisely as possible)
目前暂时无法复现
### Ⅴ. Anything else we need to know?
### Ⅵ. Environment:
- JDK version : 1.8
- OS : centos
- Others: seata的rds是5.7.28
Contributor guide
Research direction
Start with io.seata.core.store.db.LockStoreDataBaseDAO.acquireLock and follow the stack trace through DataBaseLocker and DefaultCore.branchRegister. Inspect the Apache DBCP connection-pool settings and run a batch-load scenario against the reported MySQL 5.7.28 environment. Done means the branch-registration path no longer times out waiting for an idle connection, with a regression test or documented reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100