nextflow-io / nextflow-io/nextflow
Add process directive to re-use existing task directory on resume
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
New feature
Allow automatic resubmission of processes which overrun maximum cluster time but can been checkpointed and therefore resumed.
Usage scenario
Running nextflow on a maintained cluster, maximum job durations may be 72 hours. Many genome assembly jobs will take longer, but the tools (e.g. metaBAT, metaSPADES) will save checkpoints.
Conventional resume behaviour will not help, as all running tasks will be interrupted, and the resume will recommence the failed jobs afresh.
Suggest implementation
Have an option for cluster jobs (e.g. PBSPro) to resubmit within the same working directory if a job did not run to completion.
I am currently trying to do this manually using the feedback loop, but this appears nontrivial for non-singleton inputs, and I am not sure if I can detect a time-overrun job and override the error handling.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by tracing Nextflow’s resume handling and cluster-job error handling, especially the PBSPro-style executor path mentioned in the issue. Define how a time-overrun, checkpointable task is detected and resubmitted in the same work directory, including non-singleton inputs; done requires an agreed directive and verified behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100