Handling graceful shutdown on remote workers side

Bug description
I have an application which uses remote partitioned batch jobs which are sent to the workers via JMS.
I also have ThreadPoolTaskExecutor configured on the worker side, so the chunks can be processed in parallel.
I was testing the graceful shutdown behavior on the worker side.

One of the testcase was to test what is happening when the processing time of a step on remote side takes longer than the graceful period.
The expected scenario in this case that after the graceful period expires then the partition step terminates end the step state is going to be STOPPED in the database.

In my case, the application just starts hanging, Spring is not able to fully close the spring context in this scenario. It's hanging in an endless loop in RepeatTemplate.executeInternal(). This calls TaskExecutorRepeatTemplate.getNextResult() there it tries calls runnable.expect() which calls queue.expect();. Since spring already tries to Interrupt everything this call will fail with an InterruptedException which then will be translated to a RepeatException.

https://github.com/spring-projects/spring-batch/blob/e6c27273fa2b3713c6f2d472bf3de1b18f8e5eba/spring-batch-infrastructure/src/main/java/org/springframework/batch/repeat/support/RepeatTemplate.java#L204-L217

Here couple of things can fail:

• doHandle calls DefaultExceptionHandler
  https://github.com/spring-projects/spring-batch/blob/e6c27273fa2b3713c6f2d472bf3de1b18f8e5eba/spring-batch-infrastructure/src/main/java/org/springframework/batch/repeat/exception/DefaultExceptionHandler.java#L37-L39
  This can be overridden by a custom ExceptionHandler so no NPE will be thrown.

• in case DEBUG is enabled then NPE can also be thrown here, since the unwrapped throwable is null
  https://github.com/spring-projects/spring-batch/blob/e6c27273fa2b3713c6f2d472bf3de1b18f8e5eba/spring-batch-infrastructure/src/main/java/org/springframework/batch/repeat/support/RepeatTemplate.java#L288-L290
  This can also be fixed by turning of DEBUG.

• and finally here:

https://github.com/spring-projects/spring-batch/blob/e6c27273fa2b3713c6f2d472bf3de1b18f8e5eba/spring-batch-infrastructure/src/main/java/org/springframework/batch/repeat/support/RepeatTemplate.java#L215-L217

I would expect running to be set to false, however it won't happen the RepeatContext is still not complete.

• Using reflect I was able to add an RepeatListener to RepeatTemplate which calls the context.setTerminateOnly() when the application is shutting down. This allows to break the endless loop here, but after that in AbstarctStep , it again tries to rethrow null after it extracted out the cause from this RepeateException
  https://github.com/spring-projects/spring-batch/blob/e6c27273fa2b3713c6f2d472bf3de1b18f8e5eba/spring-batch-core/src/main/java/org/springframework/batch/core/step/AbstractStep.java#L232

Environment
Please provide as many details as possible: Spring Batch version, Java version, which database you use if any, etc

• openjdk version "17.0.7" 2023-04-18 LTS
• Spring Batch 5.0.2
• Spring Boot 3.1.1
• PostrgeSQL 15.3

Steps to reproduce
See above

Expected behavior

• after the graceful period Spring shall be able to forcefully close the ApplicationContext
• no NPE or other exception is expected to be thrown.
• the related step state shall be saved using STOPPED state in the datatabase.

Minimal Complete Reproducible example
TBD, I will try to create a minimalistic example for this.
springbatchissue.zip
Steps to reproduce:

• unzip
• execute ./gradlew jibDockerBuild to create a docker image
• start the stack using docker-compose up
• check the logs for the worker, immediately after the first message is received by the worker execute kill -15 1 to kill it

You will see the app won't terminate after the graceful period ends. execute kill -3 1 and you will see that it's hanging in an endless loop

Start by reproducing the shutdown hang with the provided Docker Compose example, then inspect RepeatTemplate.java, DefaultExceptionHandler.java, and AbstractStep.java at the linked locations. Compare the interruption and termination paths against the expected graceful-shutdown behavior. Done means the worker exits after the graceful period without an NPE and the step is persisted as STOPPED.