nextflow-io / nextflow-io/nextflow
Job arrays support for HTCondor
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
New feature
Hello,
I would like to ask if the new "Job Array" functionality can also be extended to HT Condor.
The main problem we are facing currently is that the condor scheduler daemon gets overwhelmed if we submit > 10000 jobs across different nextflow pipelines running on our cluster. If these could be submitted as job arrays, this would greatly reduce the load on the scheduler.
The implementation (from the user experience side) can be the same as for other schedulers like SLURM. There will be a array <X> as an additional parameter to the executor. Nextflow then launches one jobs for ever <X> number of process instances, as an array.
Thanks,
Vivek
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the HTCondor executor entry point and compare its job submission behavior with the existing SLURM job-array support. Determine how the array <X> executor parameter should group process instances, then verify that submissions reduce scheduler load while preserving the requested user experience.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- hpc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100