Batch-level error rates for early shutdown

Priority Level

Medium

Task Summary

Our error-rate calculation for early shutdown in the ConcurrentThreadExecutor attempts to calculate error rates more or less in real time. This can lead to massive overestimates of the error rate, particularly for jobs with high concurrency, since jobs can fail faster than they succeed.

The proposal of this issue is to instead calculate error rates at the batch level (i.e., outside of ConcurrentThreadExecutor).

A couple benefits of this approach:

• The batch-level error rate will be a much more stable measurement at a consistent scale across jobs.
• This will allow the early-shutdown mechanism to be applied to generators that do not support concurrency.

One downside is that you always have to wait for at least one batch to complete. This seems acceptable given that the batch size is adjustable and (for cases where this all matters) will generally be much smaller than the target number of records.

Start by tracing early-shutdown error-rate handling through ConcurrentThreadExecutor and the surrounding batch orchestration. Determine where batch completion and adjustable batch size are handled, then move the calculation to batch level so non-concurrent generators can use it. Done means error rates are stable per batch and early shutdown waits for at least one completed batch; add or update the relevant tests found during that trace.