ebean-orm / ebean-orm/ebean

RejectionException while shutting down Ebean and NotifyOfCommit should be scheduled

Open
#2,565 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.5k
Forks
267
Avg merge
3d 20h
Merged PRs (30d)
5

Description

Application server running in Docker with active docker health check sometimes causes application server to restart (memory leaks in JVM). The restart is a regular application server shutdown and not a pure kill -9. In that case Ebean will be shut down and often throw a RejectionException during some commits that might still be executed on the server.

I am wondering if that is a race condition that might need to be fixed. I could imagine the code flow is like

  1. commit()
  2. db.shutdown()
  3. notifyCommit(), which calls an already terminated scheduler causing the exception

Maybe the exception should be handled in Ebean, if it is expected to be thrown during shutdown.

ERROR i.e.s.transaction.TransactionManager 
NotifyOfCommit failed. L2 Cache potentially not notified.
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1a6dea8d[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@370f053c[Wrapped task = io.ebeaninternal.server.executor.DefaultBackgroundExecutor$$Lambda$708/0x0000000801193800@62db3d40]] rejected from io.ebeaninternal.server.executor.DaemonScheduleThreadPool@7b14312[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8564]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2057)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:827)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:715)
	at io.ebeaninternal.server.executor.DefaultBackgroundExecutor.submit(DefaultBackgroundExecutor.java:75)
	at io.ebeaninternal.server.executor.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:80)
	at io.ebeaninternal.server.transaction.TransactionManager.notifyOfCommit(TransactionManager.java:452)
	at io.ebeaninternal.server.transaction.JdbcTransaction.notifyCommit(JdbcTransaction.java:934)
	at io.ebeaninternal.server.transaction.JdbcTransaction.postCommit(JdbcTransaction.java:1005)
	at io.ebeaninternal.server.transaction.JdbcTransaction.flushCommitAndNotify(JdbcTransaction.java:999)
	at io.ebeaninternal.server.transaction.JdbcTransaction.commit(JdbcTransaction.java:1058)
	at io.ebeaninternal.api.ScopeTrans.commitTransaction(ScopeTrans.java:140)
	at io.ebeaninternal.api.ScopedTransaction.commit(ScopedTransaction.java:110)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading TransactionManager.notifyOfCommit and the shutdown behavior of DefaultBackgroundExecutor. Trace the commit path through JdbcTransaction.postCommit, flushCommitAndNotify, and commit to confirm whether notification can reach a terminated scheduler. Done means establishing and testing behavior for commits during shutdown, with no unexpected RejectedExecutionException or lost cache notification.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.