linkedin / linkedin/dr-elephant
javax.persistence.PersistenceException: java.sql.SQLException: Connection is closed
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
I have got mr job result in UI, data is below
I looked through 28962 jobs today.
About 552 of them could use some tuning.
About 16457 of them need some serious attention!
mysql>select count() from yarn_app_heuristic_result;
+----------+
| count() |
+----------+
| 3549752 |
+----------+
But has ERROR info in dr_elephant.log every 2-3 minutes:
08-29-2018 20:11:33 ERROR [dr-el-executor-thread-0] com.linkedin.drelephant.ElephantRunner : java.sql.SQLException: Connection is closed!
08-29-2018 20:11:33 ERROR [dr-el-executor-thread-0] com.linkedin.drelephant.ElephantRunner : javax.persistence.PersistenceException: java.sql.SQLException: Connection is closed!
at com.avaje.ebeaninternal.server.transaction.JdbcTransaction.rollback(JdbcTransaction.java:641)
at com.avaje.ebeaninternal.server.transaction.JdbcTransaction.rollback(JdbcTransaction.java:622)
at com.avaje.ebeaninternal.server.core.BeanRequest.rollbackTransIfRequired(BeanRequest.java:87)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.saveRecurse(DefaultPersister.java:284)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.save(DefaultPersister.java:248)
at com.avaje.ebeaninternal.server.core.DefaultServer.save(DefaultServer.java:1568)
at com.avaje.ebeaninternal.server.core.DefaultServer.save(DefaultServer.java:1558)
at com.avaje.ebean.Ebean.save(Ebean.java:453)
at play.db.ebean.Model.save(Model.java:91)
at com.linkedin.drelephant.ElephantRunner$ExecutorJob.run(ElephantRunner.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Connection is closed!
at com.jolbox.bonecp.ConnectionHandle.checkClosed(ConnectionHandle.java:459)
at com.jolbox.bonecp.ConnectionHandle.rollback(ConnectionHandle.java:1270)
at
base info:
branch : master
hadoop_version=2.6.0-cdh5.13.0
spark_version=1.4.0
mysql : 5.7.23-linux-glibc2.12-x86_64
mysqladmin -hxxx -uroot -pxxx status
Uptime: 1124033 Threads: 6 Questions: 60480250 Slow queries: 5 Opens: 381 Flush tables: 1 Open tables: 345 Queries per second avg: 53.806
i don't understand , why threads is always 6 ?
Is “SQLException: Connection is closed” related to threads being too small?
3ks very much
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the stack trace at com.linkedin.drelephant.ElephantRunner$ExecutorJob.run in ElephantRunner.java:176 and compare it with the reported MySQL status. Trace the connection and transaction lifecycle around that save operation. Done means determining why the connection is closed and whether the thread count is related, with the result verified against the recurring log error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100