nextflow-io / nextflow-io/nextflow

Add process directive to re-use existing task directory on resume

Open
#1,728 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

lang/processes
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

New feature

Allow automatic resubmission of processes which overrun maximum cluster time but can been checkpointed and therefore resumed.

Usage scenario

Running nextflow on a maintained cluster, maximum job durations may be 72 hours. Many genome assembly jobs will take longer, but the tools (e.g. metaBAT, metaSPADES) will save checkpoints.

Conventional resume behaviour will not help, as all running tasks will be interrupted, and the resume will recommence the failed jobs afresh.

Suggest implementation

Have an option for cluster jobs (e.g. PBSPro) to resubmit within the same working directory if a job did not run to completion.

I am currently trying to do this manually using the feedback loop, but this appears nontrivial for non-singleton inputs, and I am not sure if I can detect a time-overrun job and override the error handling.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by tracing Nextflow’s resume handling and cluster-job error handling, especially the PBSPro-style executor path mentioned in the issue. Define how a time-overrun, checkpointable task is detected and resubmitted in the same work directory, including non-singleton inputs; done requires an agreed directive and verified behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.