apache / apache/incubator-seata

Cannot get a connection, pool error Timeout waiting for idle object

Open
#2,994 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
26k
Forks
8.8k
Avg merge
1d 8h
Merged PRs (30d)
4

Description

- [ ] I have searched the [issues](https://github.com/seata/seata/issues) of this repository and believe that this is not a duplicate.

### Ⅰ. Issue Description

线上环境,运行版本:1.0.0 ,这两天突然出io.seata.core.exception.TransactionException: branch register request failed. xid=xx.xx.xx.xx:8091:2336353978, msg=Cannot get a connection, pool error Timeout waiting for idle object

### Ⅱ. Describe what happened

线上环境凌晨会有批处理任务执行导致

If there is an exception, please attach the exception trace:

```
io.seata.core.exception.TransactionException: branch register request failed. xid=xxx.xxx.xxx.xxx:8091:2336353978, msg=Cannot get a connection, pool error Timeout waiting for idle object
at io.seata.server.AbstractTCInboundHandler$4.execute(AbstractTCInboundHandler.java:175)
at io.seata.server.AbstractTCInboundHandler$4.execute(AbstractTCInboundHandler.java:167)
at io.seata.core.exception.AbstractExceptionHandler.exceptionHandleTemplate(AbstractExceptionHandler.java:117)
at io.seata.server.AbstractTCInboundHandler.handle(AbstractTCInboundHandler.java:167)
at io.seata.core.protocol.transaction.BranchRegisterRequest.handle(BranchRegisterRequest.java:136)
at io.seata.server.coordinator.DefaultCoordinator.onRequest(DefaultCoordinator.java:544)
at io.seata.core.rpc.DefaultServerMessageListenerImpl.onTrxMessage(DefaultServerMessageListenerImpl.java:91)
at io.seata.core.rpc.netty.RpcServer.dispatch(RpcServer.java:295)
at io.seata.core.rpc.netty.AbstractRpcRemoting$3.run(AbstractRpcRemoting.java:384)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.seata.common.exception.StoreException: Cannot get a connection, pool error Timeout waiting for idle object
at io.seata.core.store.db.LockStoreDataBaseDAO.acquireLock(LockStoreDataBaseDAO.java:186)
at io.seata.server.lock.db.DataBaseLocker.acquireLock(DataBaseLocker.java:64)
at io.seata.server.lock.AbstractLockManager.acquireLock(AbstractLockManager.java:59)
at io.seata.server.session.BranchSession.lock(BranchSession.java:273)
at io.seata.server.coordinator.DefaultCore.lambda$branchRegister$0(DefaultCore.java:85)
at io.seata.server.session.GlobalSession.lockAndExcute(GlobalSession.java:605)
at io.seata.server.coordinator.DefaultCore.branchRegister(DefaultCore.java:70)
at io.seata.server.coordinator.DefaultCoordinator.doBranchRegister(DefaultCoordinator.java:203)
at io.seata.server.AbstractTCInboundHandler$4.execute(AbstractTCInboundHandler.java:172)
... 12 common frames omitted
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at io.seata.core.store.db.LockStoreDataBaseDAO.acquireLock(LockStoreDataBaseDAO.java:114)
... 20 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 22 common frames omitted
```

### Ⅲ. Describe what you expected to happen

### Ⅳ. How to reproduce it (as minimally and precisely as possible)

目前暂时无法复现

### Ⅴ. Anything else we need to know?

### Ⅵ. Environment:

- JDK version : 1.8
- OS : centos
- Others: seata的rds是5.7.28

Contributor guide

Open the contributing guide

Research direction

Start with io.seata.core.store.db.LockStoreDataBaseDAO.acquireLock and follow the stack trace through DataBaseLocker and DefaultCore.branchRegister. Inspect the Apache DBCP connection-pool settings and run a batch-load scenario against the reported MySQL 5.7.28 environment. Done means the branch-registration path no longer times out waiting for an idle connection, with a regression test or documented reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.