Local executor: IllegalThreadStateException in LocalTaskHandler.checkIfCompleted hard-aborts the whole session instead of just the affected task

Bug report

A running session using the local executor occasionally aborts the entire session with:

java.lang.IllegalThreadStateException: process hasn't exited
	at java.base/java.lang.ProcessImpl.exitValue(ProcessImpl.java:452)
	at nextflow.executor.local.LocalTaskHandler.checkIfCompleted(LocalTaskHandler.groovy:220)
	at nextflow.processor.TaskPollingMonitor.checkTaskStatus(TaskPollingMonitor.groovy:709)
	at nextflow.processor.TaskPollingMonitor.checkAllTasks(TaskPollingMonitor.groovy:628)
	at nextflow.processor.TaskPollingMonitor.pollLoop(TaskPollingMonitor.groovy:498)
	...

This is thrown from inside the task-monitor poll-loop thread, uncaught, and it triggers a hard Session.abort() (Session aborted -- Cause: process hasn't exited) rather than being retried or logged as a transient condition. Every still-running task at that moment is killed (SIGTERM) and the whole run ends, even though nothing had actually failed — WorkflowStats at abort time consistently shows failedCount=0.

ProcessImpl.exitValue() throwing IllegalThreadStateException normally just means "the process hasn't exited yet" — it's the expected result when a process is still legitimately running. It looks like checkIfCompleted calls it in a branch that assumes the process has already terminated (e.g. because the wrapper script's exit-code sentinel file appeared), and doesn't handle the case where the JVM's own Process handle hasn't caught up yet — a race between the external completion signal and ProcessImpl's internal reaping state, rather than a real fault.

Impact

For a long single-task run (a multi-hour download, in our case) this is a full session loss, not a single-task failure — there's no partial-progress recovery, since the task itself gets killed as a side effect of the session abort. It's happened multiple times across unrelated runs/machines, so it doesn't look tied to one host or one specific task.

Environment

• Nextflow: 26.04.6 (build 12646) — also reproduced on an earlier self-updated build before that, so this isn't fixed by picking up the latest release.
• Executor: local
• OS: Ubuntu 24.04.4 LTS, kernel 6.8.0-138-generic, x86_64
• Java: OpenJDK 21.0.12 (Nextflow's bundled/selected runtime; host default java is a separate OpenJDK 11 install)
• Container engine: both Docker and Singularity/Apptainer runs have hit it

Reproduction instances

1) Single long-running download task (most recent, full log available)

$ nextflow run erikrikarddaniel/metatdenovo -r 0439d4d47ae1ae3df27a115beb6f7e03ba884fb1 -resume -profile test,singularity --outdir results --orf_caller metaeuk --metaeuk_db_name UniRef50

(the exact commit that produced the log below, on a fork branch that reproduces this; the pipeline itself just needs to have one task that runs for an hour or more — the METAEUK_DOWNLOAD process here is otherwise unremarkable)

Task METAEUK_DOWNLOAD (UniRef50) submitted at 14:10:34.174, running normally (a wget-style download at ~43% progress, no OOM, ~4TB free disk). At 15:10:34.326 — exactly 60 minutes later — the poll loop hit the exception and aborted:

sep.-01 15:10:34.326 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=NFCORE_METATDENOVO:METATDENOVO:METAEUK:METAEUK_DOWNLOAD (UniRef50); work-dir=/data/dl/dev/nf-core-metatdenovo/work/8b/1c491aaa89510934fec49209f7818b
  error [java.lang.IllegalThreadStateException]: process hasn't exited
sep.-01 15:10:34.331 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_METATDENOVO:METATDENOVO:METAEUK:METAEUK_DOWNLOAD (UniRef50)'

Caused by:
  process hasn't exited

java.lang.IllegalThreadStateException: process hasn't exited
	at java.base/java.lang.ProcessImpl.exitValue(ProcessImpl.java:452)
	at nextflow.executor.local.LocalTaskHandler.checkIfCompleted(LocalTaskHandler.groovy:220)
	at nextflow.processor.TaskPollingMonitor.checkTaskStatus(TaskPollingMonitor.groovy:709)
	at nextflow.processor.TaskPollingMonitor.checkAllTasks(TaskPollingMonitor.groovy:628)
	at nextflow.processor.TaskPollingMonitor.pollLoop(TaskPollingMonitor.groovy:498)
	...
sep.-01 15:10:34.335 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: process hasn't exited

The task's own work directory shows exit code 143 (SIGTERM) — a side effect of the session-level abort/cleanup killing the still-running download, not the original cause.

The exactly-60-minutes gap between submission and the exception is suspicious enough to mention even though we only have one data point for it — it may point at some hourly-scale timer/threshold rather than a pure scheduling race, but we haven't been able to pin down what.

2) Mid-run abort on a long multi-task pipeline run (different machine, same signature)

A separate ~82M-contig assembly run hit the same IllegalThreadStateException/Session aborted signature after ~30h and 1852 successfully completed tasks, killing 7 still-running tasks (WorkflowStats: failedCount=0, abortedCount=7). Same fix applied (-resume) — the 1852 cached/succeeded tasks were reused, only the 7 aborted tasks re-ran.

We've now seen this at least 4 times across two machines over about a week, always with the identical stack trace/root class (ProcessImpl.exitValue → LocalTaskHandler.checkIfCompleted).

Workaround

-resume recovers cleanly when there's substantial already-cached work to fall back on — in reproduction (2) above, the 1852 cached/succeeded tasks were reused and only the 7 aborted tasks re-ran.

It does not recover a run whose in-flight task is itself long-running and has no internal resume/checkpoint capability, like the download in reproduction (1): since that task never completed, -resume just re-submits it from scratch. If the same session-abort fires again before the task finishes a second time, it aborts again, and the task restarts from zero yet again — in our case this has now happened on the same download three times running (-resume included each time), so a task whose own runtime is comparable to (or a multiple of) whatever interval triggers this may never be able to complete under Nextflow's orchestration at all. We're now trying the workaround of running that download directly on the host, outside Nextflow, and having the pipeline pick up its output via storeDir on the next -resume — not yet confirmed whether that avoids the problem.

Question

Is this a known race in LocalTaskHandler/ProcessImpl reaping? Is there a config knob to make checkIfCompleted tolerant of a transient "not exited yet" read (e.g. retry a few times before treating it as fatal) rather than aborting the whole session on the first occurrence?

Full .nextflow.log for reproduction (1) attached. Happy to try a specific Nextflow build/flag if that'd help narrow it down.

.nextflow.log

Start in nextflow.executor.local.LocalTaskHandler.groovy at checkIfCompleted around line 220, then trace how TaskPollingMonitor.checkTaskStatus handles the result. Use the attached .nextflow.log and the local-executor reproduction to observe the ProcessImpl.exitValue race. Done means a transient “process hasn't exited” condition no longer aborts the whole session or kills unaffected tasks.