maxErrorRetry does not seem to be used for K8sResponseException 500 (tunnel disconnect)

Bug report

This issue was originally raised as #5574 against NF 22.10.8 but having re-written the process for DSL2 and running with NF 24.10.2 it persists, so I'm (re-)raising a new issue here

Expected behavior and actual behavior

Nextflow receives a 500 error from the kubernetes API but it would be nice to retry (as I think the error is transient). We are using maxErrorRetry and NF 24.10.2 but this does not appear to help.

The Pod was running and actually finished successfully about 15 minutes later (as shown by the k8s Pod information of the Pod left undeleted): -

  containerStatuses:
    - name: nf-0b2286d5f91a1bb1f5a67b9c8d3ada45-c297d
      state:
        terminated:
          exitCode: 0
          reason: Completed
          startedAt: '2024-12-12T21:05:05Z'
          finishedAt: '2024-12-13T00:46:54Z'

...but the response to the (in our case a short-term) 500 error appears catastrophic.

Steps to reproduce the problem

We run our workflow and it fails with this error regularly, at almost exactly the same time each day ... 00:33:31. In the exception last night it occurred again at 00:33:31. Although we are trying to determine the cause of the underlying disconnect it would be nice if NF could comply with maxErrorRetry. If it is there's no evidence in the log.

Program output

Dec-13 00:33:31.980 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=fragment:fragment_mols (888); work-dir=/work/nextflow/0b/2286d5f91a1bb1f5a67b9c8d3ada45
  error [nextflow.k8s.client.K8sResponseException]: Request GET /api/v1/namespaces/fragmentor/pods/nf-0b2286d5f91a1bb1f5a67b9c8d3ada45-c297d/status returned an error code=500 -- error trying to reach service: tunnel disconnect
Dec-13 00:33:31.997 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'fragment:fragment_mols (888)'

Caused by:
  Request GET /api/v1/namespaces/fragmentor/pods/nf-0b2286d5f91a1bb1f5a67b9c8d3ada45-c297d/status returned an error code=500 -- error trying to reach service: tunnel disconnect


nextflow.k8s.client.K8sResponseException: Request GET /api/v1/namespaces/fragmentor/pods/nf-0b2286d5f91a1bb1f5a67b9c8d3ada45-c297d/status returned an error code=500 -- error trying to reach service: tunnel disconnect
	at nextflow.k8s.client.K8sClient.makeRequestCall(K8sClient.groovy:674)
	at nextflow.k8s.client.K8sClient.makeRequest(K8sClient.groovy:630)
	at nextflow.k8s.client.K8sClient.makeRequest(K8sClient.groovy)
	at nextflow.k8s.client.K8sClient.get(K8sClient.groovy:683)
	at nextflow.k8s.client.K8sClient.podStatus(K8sClient.groovy:319)
	at nextflow.k8s.client.K8sClient.podStatus0(K8sClient.groovy:330)
	at nextflow.k8s.client.K8sClient.podState(K8sClient.groovy:463)
	at nextflow.k8s.K8sTaskHandler.getState(K8sTaskHandler.groovy:343)
	at nextflow.k8s.K8sTaskHandler.checkIfCompleted(K8sTaskHandler.groovy:413)
	at nextflow.processor.TaskPollingMonitor.checkTaskStatus(TaskPollingMonitor.groovy:651)
	at nextflow.processor.TaskPollingMonitor.checkAllTasks(TaskPollingMonitor.groovy:573)
	at nextflow.processor.TaskPollingMonitor.pollLoop(TaskPollingMonitor.groovy:443)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
	at nextflow.processor.TaskPollingMonitor$_start_closure2.doCall(TaskPollingMonitor.groovy:318)
	at nextflow.processor.TaskPollingMonitor$_start_closure2.call(TaskPollingMonitor.groovy)
	at groovy.lang.Closure.run(Closure.java:505)
	at java.base/java.lang.Thread.run(Thread.java:829)
Dec-13 00:33:32.005 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Request GET /api/v1/namespaces/fragmentor/pods/nf-0b2286d5f91a1bb1f5a67b9c8d3ada45-c297d/status returned an error code=500 -- error trying to reach service: tunnel disconnect

Environment

• Nextflow version: 24.10.2
• Java version: openjdk 11.0.25 2024-10-15
• Operating system: Debian GNU/Linux 11 (bullseye)
• Bash version: GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)

Additional context

Running in an OpenStack kubernetes cluster (v1.30) and the following nextflow config: -

process {
  pod = [
    nodeSelector: 'informaticsmatters.com/purpose-fragmentor=yes',
    imagePullPolicy: 'Always'
  ]
}
executor {
    name = 'k8s'
    queueSize = 400
}
k8s {
    httpConnectTimeout = '120s'
    httpReadTimeout = '120s'
    maxErrorRetry = '8'
    serviceAccount = 'fragmentor'
    storageClaimName = 'work'
    storageMountPath = '/work'
}

Start with nextflow.k8s.client.K8sClient.groovy, especially makeRequestCall at line 674 and the maxErrorRetry configuration, then trace how K8sTaskHandler and TaskPollingMonitor handle pod-status failures. Reproduce or inspect the reported 500 tunnel-disconnect path and verify that transient Kubernetes errors are retried according to maxErrorRetry rather than aborting the session.