AWS Batch: expose the job-termination rate limit as real config instead of reusing executor.submitRateLimit (and an internal pool name)

New feature

executor.submitRateLimit is documented as a submission throttle, but on AWS Batch it also throttles job termination. There is no supported way to configure the two independently. The only override that works today depends on an internal thread-pool name, which is not a configuration surface anyone should have to rely on.

I would like a real config option for the termination rate, and for the existing per-executor scoping to work on this executor.

Why the two are coupled today

AwsBatchExecutor creates two ThrottlingExecutor pools and resolves the rate limit for each by passing the thread-pool name into a method that expects an executor name:

// AwsBatchExecutor.createTaskMonitor()
submitter = createExecutorService('AWSBatch-executor')
reaper    = createExecutorService('AWSBatch-reaper')

private ThrottlingExecutor createExecutorService(String name) {
    final limit = config.getExecConfigProp(name, 'submitRateLimit', '50/s') as String
    ...
}

ExecutorConfig.getExecConfigProp() looks for executor.$<name>.<prop> and otherwise falls back to the global executor.<prop>. Neither $AWSBatch-executor nor $AWSBatch-reaper is a scope any user would write, so both pools fall through to the global executor.submitRateLimit. The reaper never gets its 50/s default.

Two consequences:

1. A submitRateLimit chosen to protect SubmitJob from API throttling silently throttles TerminateJob by the same factor. Cancelling a run then takes proportionally longer, and cancellation is exactly when you are racing a SIGTERM grace period.
2. executor.$awsbatch.submitRateLimit matches neither pool and is silently ignored, so the documented per-executor scoping does not work here at all.

Measured

Twenty tasks on a real Batch queue, head process run locally so the reaper could drain fully rather than being cut off. Config executor { submitRateLimit = '50/1min' }:

Creating throttling executor with opts: ...(poolName:AWSBatch-executor, limiter:RateLimiter[stableRate=0.8qps], ...)
Creating throttling executor with opts: ...(poolName:AWSBatch-reaper,   limiter:RateLimiter[stableRate=0.8qps], ...)

With NXF_TRACE=nextflow.util.ThrottlingExecutor, each reaper thread reports how long it blocked in RateLimiter.acquire():

[AWSBatch-reaper-2] TRACE - Time spent to enforce rate=0.198812
[AWSBatch-reaper-3] TRACE - Time spent to enforce rate=1.398129
[AWSBatch-reaper-4] TRACE - Time spent to enforce rate=2.59761
[AWSBatch-reaper-5] TRACE - Time spent to enforce rate=3.796748

[AWS BATCH] cleanup = killing job <id> lines arrive 1.20 s apart, which is 0.833/s, which is 50/1min. Terminating 17 jobs took 18.2 s; the last thread sat blocked on the limiter for 18.19 s. Extrapolated to a saturated queueSize = 500, a cancellation needs about 7 minutes to issue the TerminateJob calls (about 10 minutes ignoring ThrottlingExecutor's 1.2x-per-100-successes ramp-up).

That is a real problem in practice, because AwsBatchExecutor.shutdown() waits up to 60 minutes for the reaper but the head job does not get 60 minutes. When the head job is itself an AWS Batch job being terminated, ECS sends SIGTERM and then SIGKILL after ECS_CONTAINER_STOP_TIMEOUT. On a compute environment that leaves that at the ECS default of 30 s, the drain is cut off after a few dozen terminations and the remaining jobs keep running in the queue with nothing watching them.

The workaround, and why it is not good enough

Because the lookup keys on the pool name, this does work:

executor {
    submitRateLimit = '50/1min'
    '$AWSBatch-reaper' {
        submitRateLimit = '50/s'
    }
}

Verified on 26.04.6: the submitter stays at 0.8 qps, the reaper resolves to 50.0 qps, and Time spent to enforce rate stops appearing for reaper threads entirely.

It works, but it is not something to put in documentation. It hardcodes an internal thread-pool name into user config, it is undocumented and untested, and it silently stops having any effect if the pool is ever renamed. Users cannot discover it, and there is no warning if they typo it.

Proposal

1. Stop applying submitRateLimit to the reaper. It is a submission throttle; termination should keep its 50/s default unless configured otherwise.
2. Expose the termination rate as a real option, for example aws.batch.terminateRateLimit, since it is an AWS Batch concern and belongs with the other Batch tuning knobs. executor.terminateRateLimit would also work if a generic name is preferred.
3. Resolve executor config properties using the executor name (awsbatch) rather than the thread-pool name, so executor.$awsbatch.* behaves as documented for this executor.

Worth considering separately: bounding shutdown by the grace period actually available rather than a fixed 60 minutes, and prioritising or batching terminations so cancellation is not gated on a per-request rate limit at all.

Environment

Observed on 25.10.4, reproduced on 26.04.6, and the code path is unchanged on master. Reported by a Seqera Platform Enterprise customer whose cancellations were leaving several hundred Batch jobs running and blocking their queues.

Related but distinct: #7445 (post-abort submissions orphaning jobs) and #7444 (abort-path hang). All three make cancellation leave jobs behind, by different mechanisms.

Start at AwsBatchExecutor.createTaskMonitor() and its createExecutorService() method, then trace ExecutorConfig.getExecConfigProp() to understand the current pool-name lookup. Define the supported termination-rate configuration and executor-name scoping, keeping submission and termination behavior independent. The change is done when the documented per-executor configuration applies to AWS Batch and the reaper no longer inherits submitRateLimit by default.