Exceptions during chunk commit are silently ignored

Erwin Vervaet opened BATCH-2415 and commented

A TaskletStep uses various helpers to loop over all chunks to be processed and process each of them in a
separate transaction while managing things such as skippable and retryable exceptions.
In pseudo code this whole system can be summarized as follows:

while (more chunks) { // RepeatTemplate usage in TaskletStep.doExecute
	try { // RepeatTemplate.executeInternal

		try { // StepContextRepeatCallback.doInIteration
			chunkContext = queue.poll()

			{ // TaskletStep: StepContextRepeatCallback.doInChunkContext
				try { // TransactionTemplate.execute
					{ // TaskletStep: ChunkTransactionCallback.doInTransaction
						{ // ChunkOrientedTasklet.execute
							chunk = chunkProvider.provide()
							chunkProcessor.process(chunk) [A]
							chunkContext.setComplete()
							return RepeatStatus.continueIf(!chunk.isEnd())
						}
					}
				} catch (Exception e) {
					rollback()
				}
				commit() [B]
			}

		} finally {
			if (!chunkContext.isComplete()) {
				queue.add(chunkContext)
			}
		}

	} catch (Exception e) {
		{ // SimpleRetryExceptionHandler.handleException
			if (exception is fatal) {
				throw e;
			}
		}
	}
}

This code exhibits the following type of behavior:

• If no exceptions occur during chunk processing, the chunk is marked as complete and the transaction
  commits. Since the chunk its complete it is not pushed back on the queue and the repeat loop ends normally.
• A non-fatal (i.e. retryable) exception that is thrown while processing the chunk (point [A])
  will cause the chunk not to be marked complete and the transaction to rollback. Since the chunk
  is not complete its pushed back onto the queue of chunks to process. Furthermore, since the
  exception is not fatal it will be ignored by the RepeatTemplate and a new iteration will start, actually doing the retry.

So far so good. However, we're experiencing the following case:

• No exceptions occur during chunk processing. Consequently the chunk is marked as complete.
  Now an exception is thrown during transaction commit (point [B]). Since the chunk is already marked as
  complete the chunk is not pushed back on the queue and no retry is attempted. The commit failure
  caused correct chunk transaction rollback. However, since the exception is not deemed to be fatal
  it is silently ignored by the RepeatTemplate! End result: the batch completes normally but certain chunks were silently skipped!

The exception we get is (see also BATCH-2403):

2015-08-06 13:53:04.440 DEBUG o.s.b.c.s.i.SimpleRetryExceptionHandler - Handled non-fatal exception
javax.persistence.PersistenceException: org.hibernate.exception.LockAcquisitionException: could not execute statement
	at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) ~[hibernate-entitymanager-4.2.12.Final.jar:4.2.12.Final]
	at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) ~[hibernate-entitymanager-4.2.12.Final.jar:4.2.12.Final]
	at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1316) ~[hibernate-entitymanager-4.2.12.Final.jar:4.2.12.Final]
	at org.hibernate.ejb.AbstractEntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(AbstractEntityManagerImpl.java:1510) ~[hibernate-entitymanager-4.2.12.Final.jar:4.2.12.Final]
	at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:110) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]
	at org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:53) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]
	at bitronix.tm.BitronixTransaction.fireBeforeCompletionEvent(BitronixTransaction.java:478) ~[btm-2.1.2.jar:2.1.2]
	at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:193) ~[btm-2.1.2.jar:2.1.2]
	at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:120) ~[btm-2.1.2.jar:2.1.2]
	at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1012) ~[spring-tx-3.2.9.RELEASE.jar:3.2.9.RELEASE]
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:755) ~[spring-tx-3.2.9.RELEASE.jar:3.2.9.RELEASE]
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724) ~[spring-tx-3.2.9.RELEASE.jar:3.2.9.RELEASE]
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:148) ~[spring-tx-3.2.9.RELEASE.jar:3.2.9.RELEASE]
	at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:267) ~[spring-batch-core-2.2.7.RELEASE.jar:na]
	at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:77) ~[spring-batch-core-2.2.7.RELEASE.jar:na]
	at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:368) [spring-batch-infrastructure-2.2.7.RELEASE.jar:na]
...
Caused by: org.hibernate.exception.LockAcquisitionException: could not execute statement
	at org.hibernate.dialect.MySQLDialect$1.convert(MySQLDialect.java:412) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]
...
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_05]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_05]
...

I fail to see how that can be desired behavior. Or am I just missing something here?

Affects: 2.2.7, 3.0.4

Attachments:

• debug.log (399.30 kB)
• step_succeeds_when_commit_fails.patch (4.33 kB)

4 votes, 9 watchers

Start with TaskletStep.doExecute, StepContextRepeatCallback.doInChunkContext, and RepeatTemplate.executeInternal, then review SimpleRetryExceptionHandler and the attached step_succeeds_when_commit_fails.patch. Reproduce the commit-failure path described in the issue and verify that a commit exception is not silently treated as successful completion or allowed to skip the chunk.