Flush reply queue before starting remote job step. [BATCH-2652]

Wim Veldhuis opened BATCH-2652 and commented

Currently the ChunkMessageChannelItemWriter does not clear the reply queue before it starts executing. As a result, when there are pending results on the queue from a previous failed run, the run will fail immediately.

The ChunkMessageChannelItemWriter however assumes it is the only executing instance on its queues. It however does not clear pending requests and/or replies from a previous failed or aborted run.

The job should either fail before the step actually sends out its first chunk OR it should clear the queues before starting work.

Reason to ask is that we ran into a bug in the spring implementation that always left replies on the queue. As a result, after the first run (that failed) successive runs could no longer be started. The only way out was to flush the queue manually. In our case we check to not start a second job when the first is not finished, so pending messages on the reply queue are always obsolete and should be cleared. Similarly the request queue could also be cleared.

The message we got showed there were mixed up replies, but it was not clear those were from a previous failed run.

No further details from BATCH-2652

Start by locating ChunkMessageChannelItemWriter and tracing how its request and reply queues are handled before a remote job step begins. Reproduce a failed or aborted run followed by a retry, then verify that obsolete queued messages no longer make the subsequent run fail immediately.