nextflow-io / nextflow-io/nextflow

K8s: disrupted pods with condition DisruptionTarget consume the retry budget

Open
#7,378 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

executor/k8s
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

Whenever you run a nf task using the k8s executor and your pod is disrupted (with the pod condition DisruptionTarget set), the failure can consume the task's retry limit, because Nextflow does not check for the DisruptionTarget condition when deciding whether a failure is a retryable node termination.

A more concrete example: we run NF pipelines on GKE on spot-vms, sometimes our nodes get preempted and the pod will get the DisruptionTarget condition. NF will then only retry the task if it still has retry budget.

Nextflow already does treat other node-termination causes as retryable without consuming the retry budget, i.e. when the pod no longer exists (404), or failed pods with a certain status.reason. A logical extension would be to also consider the DisruptionTarget pod condition.

A potential fix would be to modify K8sClient#podState to throw a NodeTerminationException whenever it sees pod condition DisruptionTarget (alongside the existing status.reason handling in the Failed branch). NodeTerminationException implements ProcessRetryableException so then TaskProcessor should retry this correctly without consuming retry budget.

The exit code can't be used to distinguish this case. I have seen that pre-empted pods exit with status code 137, which also occurs in legitimate failures (e.g. OOM killed). The DisruptionTarget pod condition therefore seems like the right signal to go off.

Expected behavior and actual behavior

Expected: Pod failures with DisruptionTarget don't count towards retry limits.

Actual: These disruptions do count towards the limit.

Steps to reproduce the problem

I have vibe-coded a repo with a self-contained reproduction path:
https://github.com/KasperWolsink99/nf-k8s-disruption-repro

Environment
  • Nextflow version: tested on 25.10.2 and 25.04.8, likely also on master

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with K8sClient#podState and trace the existing status.reason handling for failed pods, then read NodeTerminationException and TaskProcessor to understand retry-budget behavior. Use the linked self-contained reproduction repo to exercise a pod with the DisruptionTarget condition; done means that disruption retries without consuming the task retry limit while ordinary failures still do.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy, kubernetes
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.